Share Knowledge

Recursive Fetching with RxJS expand operator

Thursday, December 10, 2020

A couple of weeks back, I stumbled upon a thread on Twitter where they implement a use-case in different frameworks/libraries (and Angular was missing). The use-case involves a recursive fetching scenario. You can check out the React and Svelte implementations here:

I wanted to implement the same thing in Angular then it dawns on me that recursive fetching with RxJS is, well, not documented. My naive approach was retryWhen. When a condition is passed for the recursion to happen, I deliberately throw something; so retryWhen can be invoked so the my pipeline can be re-executed.

Then I learned about expand() operator and everything just makes sense. Check code snippet below.

expand(): Recursively projects each source value to an Observable which is merged in the output Observable.

Here's the Stackblitz to the Angular implementation: https://stackblitz.com/edit/angular-ivy-rxjs-expand-recursion?file=src/app/app.component.ts

Code Examples


getHN(postId: string) {
   const fetch$ = this.http.get(`https://hacker-news.firebaseio.com/v0/item/${postId}.json`)
         delay(1000), // to stimulate loading

   // expand() approach
   return fetch$.pipe(
      expand((post: any) => { // the original "post" from "postId" and every "parentPost" afterwards. This is recursive
         return post.parent // if there's "parent", we refetch, if not, we return EMPTY to complete the "expand()"
               ? this.http.get(`https://hacker-news.firebaseio.com/v0/item/${post.parent}.json`)
               : EMPTY
      reduce((finalPost, latestPost: any) => { // rxjs "reduce()" acts exactly like Array.reduce. Here, "latestPost" is each "post" from "expand()" including the original "post". reduce() will keep reducing until "expand()" completes with EMPTY
         return latestPost.title
               ? { ...finalPost, title: latestPost.title, date: new Date(d.time * 1000).toLocaleDateString("en-US") }
               : finalPost;


getFN(postId: string) {
   const fetch$ = this.http.get(`https://hacker-news.firebaseio.com/v0/item/${postId}.json`)
         delay(1000), // to stimulate loading

   // naive approach
   return fetch$.pipe(
      concatMap((post: any) => {
         // concatMap to fetch parent post or return the post
         let temp = post; // create "temp" to hold new post each time
         return temp.parent // if current "post" has parent (we need to fetch this parent recursively)
               ? defer(
                  // use defer() so "temp.parent" has the newest parent (recursive)
                  () => this.http.get(`https://hacker-news.firebaseio.com/v0/item/${temp.parent}.json`)
               ).pipe( // we "pipe" on the "defer()" so "retryWhen" will retry the whole "defer()"
                  map((parent: any) => {
                     if (parent.parent) { // again, check if there's "parent" still
                        temp = parent; // reassign "temp" so when "defer()" is retried, it has new value of "temp"
                        throw temp; // you can throw ANYTHING here
                     return { ...data, title: `re: ${parent.title}` }; // no parent, return with the title
                  retryWhen(_ => _) // this is ugly. In the "map", we throw so retryWhen is invoked but we don't need to do anything, just invoke so the whole "defer()" gets re-run
               : of(post)
      map((finalPost: any) => ({...finalPost, date: new Date(d.time * 1000).toLocaleDateString("en-US")}))
Chau Tran

Software Engineer @swimlane | Core Team @nestjs | Author @nartc/automapper

Have a question or comment?