Javascript is required
·
8 min read
·
1378 views

How I Built a Twitter Keyword Monitoring Using a Serverless Node.js Function With AWS Amplify

How I Built a Twitter Keyword Monitoring Using a Serverless Node.js Function With AWS Amplify Image

In this article, I will demonstrate to you how I built a simple serverless Node.js function on AWS that sends me a daily email with a list of tweets that mention me on Twitter.

Recently, I used Twilert and Birdspotter for that purpose, which are specialized tools for Twitter keyword monitoring. But their free plans/trials don't fulfill my simple requirements, so I decided to implement them independently.

Prerequisites

I chose again AWS Amplify to deploy the serverless function to AWS.

If you don't already have an AWS account, you'll need to create one to follow the steps outlined in this article. Please follow this tutorial to create an account.

Next, you need to install and configure the Amplify Command Line Interface (CLI).

The serverless function will need access to secrets stored in the AWS Secret Manager. My article “How to Use Environment Variables to Store Secrets in AWS Amplify Backend” will guide you through this process.

Add Serverless Function to AWS

The first step is to add a new Lambda (serverless) function with the Node.js runtime to the Amplify application.

The function gets invoked on a recurring schedule. In my case, it will be invoked every day at 08:00 PM.

Let's add the serverless function using the Amplify CLI:

bash
1 amplify add function
2? Select which capability you want to add: Lambda function (serverless function)
3? Provide an AWS Lambda function name: twittersearchfunction
4? Choose the runtime that you want to use: NodeJS
5? Choose the function template that you want to use: Hello World
6? Do you want to configure advanced settings? Yes
7? Do you want to access other resources in this project from your Lambda function? No
8? Do you want to invoke this function on a recurring schedule? Yes
9? At which interval should the function be invoked: Daily
10? Select the start time (use arrow keys): 08:00 PM
11? Do you want to enable Lambda layers for this function? No
12? Do you want to configure environment variables for this function? No
13? Do you want to configure secret values this function can access? No
14? Do you want to edit the local lambda function now? No

Get a list of tweets for a specific Twitter keyword

Now it's time to write the JavaScript code that returns a list of tweets for a given keyword.

Let's start by writing the twitter-client.js module. This module uses FeedHive’s Twitter Client to access the Twitter API. The first step is to initialize the Twitter API client and trigger the request:

1const mokkappsTwitterId = 481186762
2const searchQuery = 'mokkapps'
3const searchResultCount = 100
4
5const fetchRecentTweets = async (secretValues) => {
6  // Configure Twitter API Client
7  const twitterClient = new twitterApiClient.TwitterClient({
8    apiKey: secretValues.TWITTER_API_KEY,
9    apiSecret: secretValues.TWITTER_API_KEY_SECRET,
10    accessToken: secretValues.TWITTER_ACCESS_TOKEN,
11    accessTokenSecret: secretValues.TWITTER_ACCESS_TOKEN_SECRET,
12  })
13
14  // Trigger search endpoint: https://github.com/FeedHive/twitter-api-client/blob/main/REFERENCES.md#twitterclienttweetssearchparameters
15  const searchResponse = await twitterClient.tweets.search({
16    q: searchQuery,
17    count: searchResultCount,
18    result_type: 'recent',
19  })
20
21  // Access statuses from response
22  const statuses = searchResponse.statuses
23}

Next, we want to filter the response into three groups:

  • Tweets: Tweets from the last 24 hours that were not published by my Twitter account and are no replies or retweets
  • Replies: Tweets from the last 24 hours that were not published by my Twitter account and are replies
  • Retweets: Tweets from the last 24 hours that were not published by my Twitter account and are retweets

Let's start by the filtering the statuses response for "normal" tweets that are no replies or retweets:

1const isTweetedInLast24Hours = (status) => {
2  const tweetDate = new Date(status.created_at)
3  const now = new Date()
4  const timeDifference = now.getTime() - tweetDate.getTime()
5  const daysDifference = timeDifference / (1000 * 60 * 60 * 24)
6  return daysDifference <= 1
7}
8
9const fetchRecentTweets = async (secretValues) => {
10  // ...
11  const statuses = searchResponse.statuses
12
13  const tweets = statuses.filter((status) => {
14    const isNotOwnAccount = status.user.id !== mokkappsTwitterId
15    const isNoReply = status.in_reply_to_status_id === null
16    const isNoRetweet = status.retweeted_status === null
17    return isNotOwnAccount && isNoReply && isNoRetweet && isTweetedInLast24Hours(status)
18  })
19}

Now we can filter for retweets and replies in a similar way:

1const retweets = statuses.filter((status) => {
2  const isNotOwnAccount = status.user.id !== mokkappsTwitterId
3  const isRetweet = status.retweeted_status
4  return isNotOwnAccount && isRetweet && isTweetedInLast24Hours(status)
5})
6
7const replies = statuses.filter((status) => {
8  const isNotOwnAccount = status.user.id !== mokkappsTwitterId
9  const isReply = status.in_reply_to_status_id !== null
10  return isNotOwnAccount && isReply && isTweetedInLast24Hours(status)
11})

The last step is to map the results to a very simple HTML structure that will be rendered inside the email body:

1const { formatDistance } = require('date-fns')
2
3const mapStatus = (status) => {
4  const {
5    id_str: id,
6    created_at,
7    in_reply_to_screen_name,
8    in_reply_to_status_id_str,
9    text,
10    retweet_count,
11    favorite_count,
12    user: { screen_name: user_screen_name, followers_count, created_at: userCreatedAt, friends_count },
13  } = status
14  const createdAtLocaleString = new Date(created_at).toLocaleString()
15  const url = `https://twitter.com/${user_screen_name}/status/${id}`
16  const userUrl = `https://twitter.com/${user_screen_name}`
17  const originalUrl = in_reply_to_screen_name
18    ? `https://twitter.com/${in_reply_to_screen_name}/status/${in_reply_to_status_id_str}`
19    : null
20  const userCreatedDateDistance = formatDistance(new Date(), new Date(userCreatedAt))
21
22  return `
23    <div style="margin-bottom: 20px; padding: 10px; border: 1px solid gray; border-radius: 5px;">
24      <h2>From <a href=${userUrl}>${user_screen_name}</a> at ${createdAtLocaleString}</h2>
25      <small><strong>Followers:</strong> ${followers_count}, <strong>Following:</strong> ${friends_count}, <strong>Account Created:</strong> ${userCreatedDateDistance} ago</small>
26      <h3>${text}</h3>
27      <a href=${url} style="margin-top: 10px">Tweet</a>
28      <small style="margin-top: 5px">(<strong>Likes:</strong> ${favorite_count}, <strong>Retweets: ${retweet_count})</strong></small>
29      ${originalUrl ? `<div style="margin-top: 10px"></br><a href=${originalUrl}>Original Tweet</a></div>` : ''}
30    </div>
31    `
32}
33
34const fetchRecentTweets = async (secretValues) => {
35  // ...
36  const retweets = statuses
37    .filter((status) => {
38      const isNotOwnAccount = status.user.id !== mokkappsTwitterId
39      const isRetweet = status.retweeted_status
40      return isNotOwnAccount && isRetweet && isTweetedInLast24Hours(status)
41    })
42    .map((status) => mapStatus(status))
43}

This is the code for the whole twitter-client.js module:

1const twitterApiClient = require('twitter-api-client')
2const { formatDistance } = require('date-fns')
3
4const mokkappsTwitterId = 481186762
5const searchQuery = 'mokkapps'
6const searchResultCount = 100
7
8const mapStatus = (status) => {
9  const {
10    id_str: id,
11    created_at,
12    in_reply_to_screen_name,
13    in_reply_to_status_id_str,
14    text,
15    retweet_count,
16    favorite_count,
17    user: { screen_name: user_screen_name, followers_count, created_at: userCreatedAt, friends_count },
18  } = status
19  const createdAtLocaleString = new Date(created_at).toLocaleString()
20  const url = `https://twitter.com/${user_screen_name}/status/${id}`
21  const userUrl = `https://twitter.com/${user_screen_name}`
22  const originalUrl = in_reply_to_screen_name
23    ? `https://twitter.com/${in_reply_to_screen_name}/status/${in_reply_to_status_id_str}`
24    : null
25  const userCreatedDateDistance = formatDistance(new Date(), new Date(userCreatedAt))
26
27  return `
28    <div style="margin-bottom: 20px; padding: 10px; border: 1px solid gray; border-radius: 5px;">
29      <h2>From <a href=${userUrl}>${user_screen_name}</a> at ${createdAtLocaleString}</h2>
30      <small><strong>Followers:</strong> ${followers_count}, <strong>Following:</strong> ${friends_count}, <strong>Account Created:</strong> ${userCreatedDateDistance} ago</small>
31      <h3>${text}</h3>
32      <a href=${url} style="margin-top: 10px">Tweet</a>
33      <small style="margin-top: 5px">(<strong>Likes:</strong> ${favorite_count}, <strong>Retweets: ${retweet_count})</strong></small>
34      ${originalUrl ? `<div style="margin-top: 10px"></br><a href=${originalUrl}>Original Tweet</a></div>` : ''}
35    </div>
36    `
37}
38
39const isTweetedInLast24Hours = (status) => {
40  const tweetDate = new Date(status.created_at)
41  const now = new Date()
42  const timeDifference = now.getTime() - tweetDate.getTime()
43  const daysDifference = timeDifference / (1000 * 60 * 60 * 24)
44  return daysDifference <= 1
45}
46
47const fetchRecentTweets = async (secretValues) => {
48  const twitterClient = new twitterApiClient.TwitterClient({
49    apiKey: secretValues.TWITTER_API_KEY,
50    apiSecret: secretValues.TWITTER_API_KEY_SECRET,
51    accessToken: secretValues.TWITTER_ACCESS_TOKEN,
52    accessTokenSecret: secretValues.TWITTER_ACCESS_TOKEN_SECRET,
53  })
54
55  const searchResponse = await twitterClient.tweets.search({
56    q: searchQuery,
57    count: searchResultCount,
58    result_type: 'recent',
59  })
60
61  const statuses = searchResponse.statuses
62
63  const tweets = statuses
64    .filter((status) => {
65      const isNotOwnAccount = status.user.id !== mokkappsTwitterId
66      const isNoReply = status.in_reply_to_status_id === null
67      const isNoRetweet = status.retweeted_status === null
68      return isNotOwnAccount && isNoReply && isNoRetweet && isTweetedInLast24Hours(status)
69    })
70    .map((status) => mapStatus(status))
71
72  const retweets = statuses
73    .filter((status) => {
74      const isNotOwnAccount = status.user.id !== mokkappsTwitterId
75      const isRetweet = status.retweeted_status
76      return isNotOwnAccount && isRetweet && isTweetedInLast24Hours(status)
77    })
78    .map((status) => mapStatus(status))
79
80  const replies = statuses
81    .filter((status) => {
82      const isNotOwnAccount = status.user.id !== mokkappsTwitterId
83      const isReply = status.in_reply_to_status_id !== null
84      return isNotOwnAccount && isReply && isTweetedInLast24Hours(status)
85    })
86    .map((status) => mapStatus(status))
87
88  return {
89    tweets,
90    retweets,
91    replies,
92  }
93}
94
95module.exports = fetchRecentTweets

Serverless function code

We can now use the twitter-client.js in our serverless function:

1const AWS = require('aws-sdk')
2const nodemailer = require('nodemailer')
3const fetchRecentTweets = require('./twitter-client')
4
5const secretsManager = new AWS.SecretsManager()
6const responseHeaders = {
7  'Content-Type': 'application/json',
8}
9
10exports.handler = async (event) => {
11  console.log(`👷 Function is ready to search for tweets`)
12
13  const secretData = await secretsManager.getSecretValue({ SecretId: 'YOUR_SECRET_ID' }).promise()
14  const secretValues = JSON.parse(secretData.SecretString)
15
16  const transporter = nodemailer.createTransport({
17    service: secretValues.MAIL_HOST,
18    auth: {
19      user: secretValues.MAIL_USER,
20      pass: secretValues.MAIL_PW,
21    },
22  })
23
24  const defaultMailOptions = {
25    from: secretValues.MAIL_USER,
26    to: secretValues.MAIL_SUCCESS,
27    subject: `[Mokkapps API] Twitter Search Results`,
28  }
29
30  try {
31    // Fetch recent tweets
32    const { tweets, replies, retweets } = await fetchRecentTweets(secretValues)
33
34    // Skip sending email if we have no results
35    if (tweets.length === 0 && replies.length === 0 && retweets.length === 0) {
36      return {
37        statusCode: 200,
38        headers: responseHeaders,
39        body: [],
40      }
41    }
42
43    // Send email
44    await transporter.sendMail({
45      ...defaultMailOptions,
46      html: `
47        <h1>Tweets that mentioned "mokkapps" in the last 24 hours</h1>
48        ${tweets.length === 0 ? '<p>No results</p>' : tweets.join('')}
49        <h1>Replies that mentioned "mokkapps" in the last 24 hours</h1>
50        ${replies.length === 0 ? '<p>No results</p>' : replies.join('')}
51        <h1>Retweets that mentioned "mokkapps" in the last 24 hours</h1>
52        ${retweets.length === 0 ? '<p>No results</p>' : retweets.join('')}
53      `,
54    })
55
56    return {
57      statusCode: 200,
58      headers: responseHeaders,
59      body: JSON.stringify({ tweets, replies, retweets }),
60    }
61  } catch (e) {
62    console.error('☠ Twitter Search Function Error:', e)
63    return {
64      statusCode: 500,
65      headers: responseHeaders,
66      body: e.message ? e.message : JSON.stringify(e),
67    }
68  }
69}

At this point, we can publish our function by running:

bash
amplify push

If we successfully pushed the function to AWS, we can manually invoke the function in AWS Lamba by clicking the "Test" button:

AWS Lambda Function Test

The serverless function should then send an email with a list of tweets if someone mentioned the monitored keyword in the last 24 hours:

Email sent from serverless Node.js function

Conclusion

I had a lot of fun building this simple serverless function to monitor keywords on Twitter.

Serverless functions are a perfect choice for such a monitoring tool, as we only have to pay for the execution time of the serverless function.

What do you think about my solution? Leave a comment and tell me how you monitor your Twitter keywords.

If you liked this article, follow me on Twitter to get notified about new blog posts and more content from me.

Alternatively (or additionally), you can also subscribe to my newsletter.

I will never share any of your personal data. You can unsubscribe at any time.

If you found this article helpful.You will love these ones as well.
How I Replaced Revue With a Custom-Built Newsletter Service Using Nuxt 3, Supabase, Serverless, and Amazon SES Image

How I Replaced Revue With a Custom-Built Newsletter Service Using Nuxt 3, Supabase, Serverless, and Amazon SES

Track Twitter Follower Growth Over Time Using A Serverless Node.js API on AWS Amplify Image

Track Twitter Follower Growth Over Time Using A Serverless Node.js API on AWS Amplify

Build and Deploy a Serverless GraphQL React App Using AWS Amplify Image

Build and Deploy a Serverless GraphQL React App Using AWS Amplify

The 10 Favorite Features of My Developer Portfolio Website Image

The 10 Favorite Features of My Developer Portfolio Website