From SMS to AI: Lessons from 5 Years of Chatbot Development for Social Impact

one · Other · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: social-impact chatbot engagement

ONE needed to decide whether Messenger and WhatsApp chatbots could outperform earlier mobile channels for African advocacy engagement, repeat actions, supporter updates, and long-term relationship building while staying affordable and usable for target audiences.

Constraints:

• ONE's chatbots were not AI-backed yet and used decision trees with some keyword recognition
• re-engaging users through Meta channels is constrained by 24-hour windows, paid message types, WhatsApp approval, and broadcast throttling
• user acquisition costs differed sharply between Messenger and WhatsApp and by gender
• the chatbot had to support multiple markets and languages while remaining easy for users
• stretched nonprofit teams needed reporting and cross-organizational ownership to sustain the platform

Architecture

ONE built two rule-based chatbot services: a Messenger bot on its global Facebook page and a pan-African bot available through Messenger, a website widget, and WhatsApp. The backend combines Turn as WhatsApp Business Provider, RapidPro as chat-flow builder, ActionKit for CRM supporter numbers, and Tableau on Google BigQuery for analytics. ONE iterated from focused campaign/market MVPs toward localized English and French content, re-engagement nudges, activist-group joins, petition signing, quizzes, and reporting infrastructure for meaningful performance data.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  messenger-chatbot["Messenger chatbot"] --> rapidpro-flow-builder["RapidPro flow builder"]
  messenger-chatbot --> actionkit-crm["ActionKit CRM"]
  pan-african-chatbot["Pan-African chatbot"] --> rapidpro-flow-builder
  pan-african-chatbot --> turn-whatsapp-provider["Turn WhatsApp provider"]
  pan-african-chatbot --> actionkit-crm
  pan-african-chatbot --> re-engagement-system["Re-engagement system"]
  rapidpro-flow-builder --> messenger-chatbot
  rapidpro-flow-builder --> pan-african-chatbot
  analytics-warehouse["Analytics warehouse"] --> feedback-collection["Feedback collection"]
  feedback-collection --> analytics-warehouse
  feedback-collection --> content-localization-loop["Content localization loop"]
  content-localization-loop --> rapidpro-flow-builder
  re-engagement-system --> analytics-warehouse
  activist-group-handoff["Activist group handoff"] --> pan-african-chatbot

Components

Data flow

A supporter enters through Messenger, WhatsApp, a Meta ad, email click-through, or the ONE Africa website widget. The decision-tree chatbot presents localized content, buttons, keyword-recognition paths, petitions, quizzes, FAQs, supporter sign-up, or activist-group options. RapidPro manages flows, Turn connects WhatsApp delivery, ActionKit stores supporter numbers, and chatbot usage data flows into BigQuery/Tableau analytics. Feedback polls, open questions, WhatsApp groups, and backend data inform the next iteration.

Why this architecture

ONE chose chatbots because SMS and IVR re-engagement were expensive, Messenger preference had caught up with SMS, and chat apps offered a direct conversational channel where users already spent data. The architecture remained simple and on-rails because the goal was learning, user engagement, and measurable campaign action rather than sophisticated AI automation.

Models used: decision-tree chatbot · keyword recognition · RapidPro chat flows · Turn WhatsApp Business Provider · ActionKit CRM · Google BigQuery · Tableau

Scale

• ONE had used mobile technologies for audience reach since 2012.
  “ONE had been using mobile technologies to reach new audiences since 2012”article, stated
• ONE had a mobile supporter base of 2 million users before exploring chatbots.
  “ONE, who had by this point accrued a mobile supporter base of 2 million users”article, stated
• The chatbot journey began in spring 2019 and had five iterations over five years.
  “ONE set out to design and test a chatbot in the Spring of 2019. 5 years later, ONE has produced 5 iterations on its first MVP”article, stated
• ONE currently runs two chatbots.
  “Today, ONE runs 2 chatbots.”article, stated
• The pan-African bot reaches over 38,000 unique users.
  “It currently reaches over 38,000 unique users in Kenya, Nigeria, Ethiopia, South Africa, Senegal, and DRC.”article, stated
• The pan-African bot operates across six named African countries.
  “Kenya, Nigeria, Ethiopia, South Africa, Senegal, and DRC.”article, stated
• More than 17,000 actions have been taken through the chatbot.
  “Over 17,000 actions have been taken via the chatbot, with most of those engaging taking multiple actions.”article, stated
• An email re-engagement experiment had an 8.5% click-through rate and 3.4% petition conversion via chatbot.
  “ONE achieved a CTR of 8.5%, and 3.4% of those emailed went on to sign a petition via the chatbot.”article, stated
• One Meta campaign acquired Messenger users at $0.17 and WhatsApp users at $1.77.
  “one of the best performing campaigns cost $0.17 per new Messenger user, compared to $1.77 for new WhatsApp users.”article, stated
• Only 14% of the user base is on WhatsApp despite the extra cost of running WhatsApp service.
  “only 14% of our userbase is on WhatsApp, despite the additional cost required to run a WhatsApp-based service.”article, stated
• Female users were 2.5x more expensive to acquire than male users.
  “It was also cheaper and easier to convert male users, with female users 2.5x more expensive to acquire”article, stated
• WhatsApp broadcast messages are throttled in tiers of 1-100k messages.
  “broadcast messages are restricted by a system which throttles the number of (unsolicited) messages sent to users in tiers of 1-100k messages.”article, stated
• A re-engagement nudge led 25% of nudged users to take at least one additional action.
  “data shows that 25% of users nudged using this mechanism take at least 1 additional action within the bot”article, stated

Alternatives considered (roads not taken)

continue relying mainly on SMS and IVR for re-engagement

Rejected because: ONE found regular re-engagement through SMS or IVR prohibitively expensive for a 2 million-user mobile supporter base.

build a large, feature-rich chatbot immediately

Rejected because: ONE recommends starting small because even minimal focused functionality required significant learning across flows, content, integration, testing, tracking, marketing, cost, and scalability.

assume users will discover the chatbot organically

Rejected because: Digital tools still require paid or intentional acquisition mechanisms, and chatbot acquisition costs differed sharply by channel.

run a complex automated re-engagement system

Rejected because: Meta broadcast rules and hard-to-interpret re-engagement data made the larger system too complex, so ONE scaled it back.

treat the chatbot as a discrete comms output owned by a small technical team

Rejected because: Operating in a silo made performance data, accountability, continuous improvement, and cross-team value harder to sustain.

Patterns

eval-driven-development (quality)

ONE iterated the chatbot through user research, prototype launches, user testing, polls, open questions, backend data, and feedback loops before expanding functionality and markets.

observability-tracing (operations)

The chatbot system produces usage data in BigQuery/Tableau and relies on reporting infrastructure focused on meaningful metrics rather than vanity measures.

data-quality-loop (feedback)

User feedback from WhatsApp groups, polls, open questions, testing, and backend data guided message length, small-talk handling, personalization, and language choices.

canary-rollout (release)

ONE started with focused offerings around one campaign and market at a time, then expanded toward the ONE Africa chatbot after learning from early versions.

cost-aware-routing (efficiency)

ONE compared Messenger, WhatsApp, SMS, IVR, email, paid ads, and re-engagement options by acquisition cost, channel restrictions, and retention value.

human-in-the-loop-gate (research)

ONE relied on target-audience consultation, youth activists, WhatsApp groups, and internal chatbot champions rather than backend metrics alone.

Tradeoffs

Initial chatbot scope

Chose focused campaign and market MVPs over broad multi-market chatbot launch.

Rationale: A small scope let ONE learn user behavior, flow design, content localization, integrations, data tracking, marketing, cost, and scalability before expanding.

Channel economics

Chose measure and optimize across Messenger, WhatsApp, email, ads, SMS, and IVR over assuming chatbots are automatically cheaper than prior mobile channels.

Rationale: Messenger acquisition was far cheaper than WhatsApp, but paid marketing and Meta ecosystem constraints still shaped costs.

Re-engagement design

Chose limited 24-hour nudge plus Messenger broadcasts over complex multi-week automated re-engagement across all channels.

Rationale: The broader system was difficult to operate around broadcast rules and hard-to-interpret re-engagement data, especially for WhatsApp.

Evidence for investment

Chose meaningful ROI data and reporting infrastructure over vanity engagement metrics.

Rationale: Nonprofit teams with limited budgets need to know whether chatbots create repeat actions, retention, and cross-team value, not only whether users opened the bot.

Future AI adoption

Chose retain safe, basic channels while exploring generative AI over replacing on-rails chatbots, SMS, IVR, and real-world interactions with GenAI everywhere.

Rationale: AI may improve content generation, but connectivity, cost, gendered access, phone credit, battery, and users' desire for human connection still matter.

Gotchas (what bit them)

A simple chatbot still carried a steep implementation learning curve.

Root cause: Even minimal decision-tree functionality required flow design, writing, localization, third-party integrations, data tracking, marketing, cost analysis, and scalability work.

Resolution: Start small, proceed slowly, and treat the first versions as learning vehicles rather than target-driven launches.

Backend analytics alone could not explain user experience.

Root cause: Data showed what users did, but qualitative feedback was needed to understand why they reacted that way.

Resolution: Pair backend metrics with WhatsApp groups, polls, open questions, and repeated user testing.

Chatbot acquisition required real marketing budget.

Root cause: Users did not automatically discover the tool, and Meta channel differences affected conversion and cost.

Resolution: Plan paid acquisition and compare channel economics before assuming chatbot cost savings.

Cross-app compatibility caused acquisition attrition.

Root cause: Users clicking links may not have Messenger or WhatsApp installed on the device used to open the email.

Resolution: Measure the drop-off between link clicks and conversation starts and account for channel compatibility in acquisition planning.

Meta's re-engagement rules limited retention strategy.

Root cause: Outside the free 24-hour window, messages had paid use-case restrictions, WhatsApp sign-off, and broadcast throttling.

Resolution: Scale back to one nudge within 24 hours and Messenger broadcast re-engagement where practical.

The chatbot operated in an organizational silo.

Root cause: Experimental work was handled by a small technical group and external experts rather than being integrated into day-to-day campaign and policy work.

Resolution: Integrate the chatbot across teams and appoint an internal champion responsible for adoption.

Generative AI may not be appropriate for every connectivity context.

Root cause: AI products exchange more data, can drain phone batteries faster, and may be difficult where phone credit is precious.

Resolution: Keep simpler chatbots, SMS, IVR, and real-world interactions in the channel mix for lower-connectivity or constrained users.

Evaluation

ONE evaluated the chatbot through user research, prototype testing by campaign and market, WhatsApp-group qualitative feedback, polls, open questions, backend usage analytics, acquisition experiments, re-engagement tests, reporting infrastructure, and comparison to other digital channels.

Metrics: unique users reached · actions taken · repeat actions · email click-through rate · petition conversion rate · cost per new Messenger user · cost per new WhatsApp user · WhatsApp share of user base · gendered acquisition cost ratio · link-click to conversation-start drop-off · 24-hour nudge additional-action rate · bounce and drop-off rates · ROI versus vanity metrics

Outcomes

• ONE's chatbots reached and engaged citizens across Africa through Messenger and WhatsApp.
• The chatbot drove more than 17,000 actions, with most engaged users taking multiple actions.
• User testing showed a significant majority of connected users could access and enjoy the chatbots.
• The channel helped ONE understand African audiences and feed insights into campaign, policy, and supporter-journey discussions.
• Cost-effectiveness and retention remained unresolved compared with other channels.

ONE reports two current chatbots, over 38,000 unique users across Kenya, Nigeria, Ethiopia, South Africa, Senegal, and DRC, more than 17,000 actions, Messenger acquisition as low as $0.17 per user, WhatsApp acquisition at $1.77 per user, and a 25% additional-action rate from one re-engagement nudge.