Paging Library for Android With Kotlin: Creating Infinite Lists

In this tutorial, you will build up a simple Reddit Clone that loads pages of information gradually into an infinite list using the Paging library and Room.

Version

  • Kotlin 1.2, Android 4.4, Android Studio 3

It’s no surprise that lists are an essential part of mobile development. A considerable amount of the information that you want to show users is best conveyed in a list containing similar pieces of information. Think of the news feed on your favorite app, your stream of pictures on Instagram or the endlessly scrolling content of Reddit — lists are everywhere. In this tutorial, you’ll see how the new Android Paging library can significantly improve your lists.

Understanding Infinitely Loading Lists

Sometimes, the lists that you want to display have a fixed size; there’s often just too much information to display all at once. Think about your Facebook feed — if, every time you opened the Facebook app, it pulled down and displayed every post, image or Like you or any of your friends had ever made, the app would never finish loading that list.

Instead, Facebook, and most apps with a considerable dataset, use infinitely loading lists.

Infinitely loading lists only download as much data as the user will immediately interact with and then load more data on demand as the user scrolls farther in the list.

In this tutorial, you will build up a simple Reddit Clone that pages information into an infinite list using Android Paging library.

Infinitely loading lists come with a ton of advantages:

  • The app only needs to download one or two pages to show.
  • The user gets content faster.
  • Uses far less memory.
  • Uses less data as it doesn’t need the full dataset.
  • Even during data updates and refreshes, the app responds quickly.
  • You can observe and update data more easily.
  • You can use placeholders to indicate if new content is being downloaded.

Sounds like a win-win!

Infinite Lists Without the Paging Library

Before the Paging library, utilizing progressively loading lists was a bit tricky. You’d usually begin by using a RecyclerView and listening for scroll events to figure out if the user was approaching the bottom of the list. Typically, that meant adding an OnScrollListener to the RecyclerView and doing some error-prone math to figure out if the user was close to the end of the list. This approach had several downsides:

  • It was brittle. If the user scrolled up and down quickly, they might end up double loading new content, so you had to keep track of the state to make sure you were not already loading new information.
  • It was error-prone. If you did not manage the state logic previously mentioned correctly, you could easily end up either not loading anything or loading two copies of the next page of data.

Not fun!

Getting Started

To help facilitate the process of implementing an infinitely loading list, Google introduced the Paging library. It assists you in creating progressively loading lists by providing several core classes that abstract the logic behind when to load new items. It also provides tie-ins to other core architecture components, including the LiveData streaming library and the Room storage library, which you’ll read about later.

Start by downloading the materials for this tutorial using the Download materials button at the top or bottom of this tutorial. Fire up Android Studio 3.1.3 or later and import the starter project.

You’ll see that there’s a networking package that contains several model classes and a RedditService class that provides the Reddit API. There are also several empty classes that you will flesh out throughout this tutorial.

Run the project. You should see an empty screen.

Paging Library Reddit Clone

The Reddit API returns pages of Reddit posts, so this is a perfect candidate to use the Paging library!

Choosing a DataSource

A DataSource is a class that manages the actual loading of data for your list. It provides callbacks that instruct you to load some data at a certain range or with a certain key value. There are three types of DataSource provided by the Paging library:

  • ItemKeyedDataSource
  • PageKeyedDataSource
  • PositionalDataSource

It’s important to choose the data source that best fits your structure. Use:

  • ItemKeyedDataSource if you need to use data from item N to fetch item N+1. For example, for threaded comments in a discussion app, you might need to pass the ID of the last comment to get the contents of the next comment.
  • PageKeyedDataSource if pages you load embed next/previous keys. For example, if you’re fetching social media posts, you may need to pass a next page key to load the subsequent posts.
  • PositionalDataSource if you need to fetch pages of data from a specific location in your data store. For example, you might want to request items 100 to 120.

Adding the DataSource

Your first task is to create a DataSource that will pull down Reddit information. The Reddit API returns data in such a way that you receive a one-page key that corresponds to a list of Reddit posts. Since you have one key per page of Reddit data, you should use PageKeyedDataSource.

Open up the RedditDataSource file. You should see a DataSource shell that looks like this:

class RedditDataSource : PageKeyedDataSource<String, RedditPost>() {

  override fun loadInitial(
      params: LoadInitialParams<String>,
      callback: LoadInitialCallback<String, RedditPost>) {

    TODO("not implemented")
  }

  override fun loadAfter(
      params: LoadParams<String>,
      callback: LoadCallback<String, RedditPost>) {

    TODO("not implemented")
  }

  override fun loadBefore(
      params: LoadParams<String>,
      callback: LoadCallback<String, RedditPost>) {

    TODO("not implemented")
  }
}

You’re using a PageKeyedDataSource as previously mentioned. The type of the key will be a String, and you want to produce RedditPost objects, so you must use the corresponding type values.

This class is looking pretty bare — time to start fleshing it out!

Add the following line below the class header:

private val api = RedditService.createService()

You are creating a reference to the RedditService class to download posts from the Reddit API.

Next, you need to build out the loadInitial(params:, callback:) method. Replace the TODO comment with the following code:

api.getPosts(loadSize = params.requestedLoadSize)
    .enqueue(object : Callback<RedditApiResponse> {

      override fun onFailure(call: Call<RedditApiResponse>?, t: Throwable?) {
        Log.e("RedditDataSource", "Failed to fetch data!")
      }

      override fun onResponse(
          call: Call<RedditApiResponse>?,
          response: Response<RedditApiResponse>) {

        val listing = response.body()?.data
        val redditPosts = listing?.children?.map { it.data }
        callback.onResult(redditPosts ?: listOf(), listing?.before, listing?.after)
      }
    })

First, you call through to the API passing in the requestedLoadSize that was received in the LoadInitialParams object. Then, you execute the getPosts method and provide a network callback for a success and a failure.

In the onFailure block of the network callback, you simply log an error. For a production application, you may want to indicate to the user that something went wrong.

In the onResponse block, you extract the RedditPost objects from the API response and pass them through to the callback object supplied to you. If you didn’t receive any RedditPosts from the API, simply pass in an empty list. You also pass through both the before and after keys that the Reddit API gave you.

Next, you’ll fill in the loadAfter(params:, callback:) method. Again, replace the corresponding TODO comment with:

api.getPosts(loadSize = params.requestedLoadSize, after = params.key)
    .enqueue(object : Callback<RedditApiResponse> {
      
      override fun onFailure(call: Call<RedditApiResponse>?, t: Throwable?) {
        Log.e("RedditDataSource", "Failed to fetch data!")
      }

      override fun onResponse(
          call: Call<RedditApiResponse>?,
          response: Response<RedditApiResponse>) {

        val listing = response.body()?.data
        val items = listing?.children?.map { it.data }
        callback.onResult(items ?: listOf(), listing?.after)
      }
    })

This method also looks pretty familiar. The only difference here is that, in the LoadCallback object, you’re only passing the after key.

Finally, you can finish the loadBefore(params:, callback:) method:

api.getPosts(loadSize = params.requestedLoadSize, before = params.key)
    .enqueue(object : Callback<RedditApiResponse> {

      override fun onFailure(call: Call<RedditApiResponse>?, t: Throwable?) {
        Log.e("RedditDataSource", "Failed to fetch data!")
      }

      override fun onResponse(
          call: Call<RedditApiResponse>?,
          response: Response<RedditApiResponse>) {

        val listing = response.body()?.data
        val items = listing?.children?.map { it.data }
        callback.onResult(items ?: listOf(), listing?.before)
      }
    })

The only difference this time is that you’re passing the before key rather than the after key, since this method loads previous items from the API. Pretty simple! The next component you want to utilize is your very own PagedListAdapter.

Using a PagedListAdapter

Navigate over to the RedditAdapter class. Check that the class is extending PagedListAdapter().

The first type parameter is RedditPost. That’s the model class this adapter will work with and the class the RedditDataSource produces! See? It’s all coming full circle.

The second type parameter is the RedditViewHolder, just like a normal RecyclerView.Adapter. You also need a DiffUtil that you’ll implement next.

DiffUtil is a utility class that helps to streamline the process of sending a new list to a RecyclerView.Adapter. It can generate callbacks that communicate with the notifyItemChanged or notifyItemInserted methods of the adapter to update its items in an efficient manner, without you having to deal with that complex logic.

Open RedditDiffUtilCallback. Right now, it’s just an empty shell that you’ll fill in.

There are only two methods you need to worry about — areItemsTheSame and areContentsTheSame. areItemsTheSame asks whether two items represent the same, equal object. This is usually done by comparing the unique id of the two items.

areContentsTheSame asks whether the content of the same item has changed. It means areContentsTheSame is called only after areItemsTheSame returns true. If areContentsTheSame returns false, the adapter will re-render the item because its content has changed.

Replace the areItemsTheSame method with the following code:

  return oldItem?.key == newItem?.key

Do the same with the areContentsTheSame method:

  return oldItem?.title == newItem?.title
      && oldItem?.score == newItem?.score
      && oldItem?.commentCount == newItem?.commentCount

Two RedditPosts are the same post if they have the same key. The content of a post is changed when its title, score or comment count is changed.

To learn more about DiffUtil, check out the intermediate recycler view tutorial: Intermediate RecyclerView Tutorial With Kotlin | Ray Wenderlich

Go back to RedditAdapter.

Update the onBindViewHolder method. Replace // nothing yet! comment with:

  val item = getItem(position)
  val resources = holder.itemView.context.resources
  val scoreString = resources.getString(R.string.score, item?.score)
  val commentCountString = resources.getString(R.string.comments, item?.commentCount)
  holder.itemView.title.text = item?.title
  holder.itemView.score.text = scoreString
  holder.itemView.comments.text = commentCountString

This is just a normal onBindViewHolder method with one exception — you’re using the getItem method provided by PagedListAdapter to get the RedditPost.

Wiring Them All Together

You’ve got a DataSource and a PagedListAdapter; the last thing to do is to wire them together and get a PagedList!

Navigate to MainActivity. Add to the initializeList() method the following code:

  //1
  val config = PagedList.Config.Builder()
      .setPageSize(30)
      .setEnablePlaceholders(false)
      .build()

  //2
  val liveData = initializedPagedListBuilder(config).build()

  //3
  liveData.observe(this, Observer<PagedList<RedditPost>> { pagedList ->
    adapter.submitList(pagedList)
  })

There are a few things to unpack, here.

  1. To create a PagedList, you need a PagedList.Config value to pass through a few options. The first thing this config controls is how many items the DataSource will attempt to fetch. You can use the setPageSize method to set the page size to 30 items.

    The second thing the config controls is the setEnablePlaceholders flag. You can tell the Paging library to use null values for content that hasn’t been loaded yet — as long as you know the total number of items that you will eventually need to fetch. Using placeholders has the benefit of keeping the scrollbar from jumping around as you load more content. However, since you don’t know the total number of items the Reddit API can serve, set this to false.

  2. Do not worry when you see initializedPagedListBuilder method highlighted in red because you will create this method soon. This method takes in the config you defined above and returns LivePagedListBuilder to build the LiveData.
  3. You’re observing the LiveData and submitting the value it emits via the submitList method. submitList is another special PagedListAdapter method that feeds a PagedList into the adapter and automatically starts the loading process.

    The PagedListAdapter is now receiving a PagedList, which, in turn, is calling into the DataSource to generate new items.

Next, add the initializedPagedListBuilder method after initializeList():

private fun initializedPagedListBuilder(config: PagedList.Config):
    LivePagedListBuilder<String, RedditPost> {

  val dataSourceFactory = object : DataSource.Factory<String, RedditPost>() {
    override fun create(): DataSource<String, RedditPost> {
      return RedditDataSource()
    }
  }
  return LivePagedListBuilder<String, RedditPost>(dataSourceFactory, config)
}

The Paging library exposes this handy LivePagedListBuilder class that takes in a DataSource.Factory and a PagedList.Config and produces a LiveData.

To create a LivePagedListBuilder, you need to pass it a DataSource.Factory object that the Paging library can use to construct the DataSource. Passing a factory in allows the Paging library to reconstruct the DataSource if you need to invalidate the existing one.

Since the RedditDataSource class doesn’t take any parameters, you simply call the constructor. If you were writing a production app, you might want to pass the RedditApi through to satisfy a dependency injection architecture.

A PagedList is just a modified list. It integrates with a DataSource to provide content as items in the list get consumed. As whoever is accessing items on the PagedList begins to approach the bottom of the list, it will delegate off to its DataSource to fetch new items.

You may be asking, “Why do I need a LiveData of PagedList? Why can’t I just get one PagedList?” That’s a great question.

First, it’s worth giving a small refresher on LiveData. LiveData is an architecture library provided by Google that allows you to easily implement the observer pattern, as well as stream data in a lifecycle-aware manner throughout your app.

You can find a deeper dive into the LiveData library here: Android Architecture Components: LiveData | Ray Wenderlich

The reason that you need a LiveData stream of PagedList objects is that you may want to invalidate the existing DataSource and start from scratch. For example, if you implement pull-to-refresh in your app, you may want to throw everything away and start again. By creating a LiveData of PagedList, you don’t need to do anything special after you call invalidate on the DataSource. The LiveData will simply emit a new copy of PagedList that you’ll feed into the adapter, and everything will automagically work.

Paging Library Magic

If you run the app, you should see a screen that looks something like this (albeit with different content):

Paging Library  First Round

Wahoo! The list is populating. Try scrolling down, and you will see that new content is loading as you approach the bottom of the list.

Magical!

Breaking Free of the Network Dependency

The app is firing on all cylinders, but there’s one problem — it’s entirely dependent on the network to run. If you have no internet access, the app is doomed!

You might say the app is a bit too constricted and you need to make it roomier. (Get it? Roomier? Because you’re going to use the Room database library for caching? I know, I know. Why am I writing tutorials when I should undoubtedly be a stand-up comedian?)

Note: Room is a storage library created by Google as part of the Android architecture components. It offers a flexible and efficient API for all of your local storage needs on Android. The Room library uses SQLite under the hood and allows you to write custom queries that can be executed to return model objects. It handles serializing and deserializing your model objects, so you don’t need to worry about ContentValues anymore. It also allows you to specify the return type for many of your queries.

First things first: Open the RedditDb class. It’s pretty empty right now — all it contains is a companion object to create an instance of the database.

Open the file called RedditPostDao. Add the following:

@Dao
interface RedditPostDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  fun insert(posts : List<RedditPost>)

  @Query("SELECT * FROM RedditPost")
  fun posts() : DataSource.Factory<Int, RedditPost>
}

For the most part, this is a standard Dao object in Room. However, there’s one unique thing here that you may not have seen before.

Look at the posts method:

@Query("SELECT * FROM RedditPost")
fun posts() : DataSource.Factory<Int, RedditPost>

The return type for the posts method is DataSource.Factory. That’s right, Room can actually create DataSource for you! You don’t need to write any code to create the DataSourceRoom will generate one that uses an Int key to pull RedditPost objects from the database as you scroll.

Open the file RedditDb and add a new method after the companion object closing brace to fetch the new RedditPostDao:

abstract fun postDao(): RedditPostDao

Head back to MainActivity. Replace the code in initializedPagedListBuilder(). You also need to change the return type in the method definition from LivePagedListBuilder<String, RedditPost> to LivePagedListBuilder<Int, RedditPost>. It should finish like this:

private fun initializedPagedListBuilder(config: PagedList.Config):
    LivePagedListBuilder<Int, RedditPost> {

  val database = RedditDb.create(this)
  val livePageListBuilder = LivePagedListBuilder<Int, RedditPost>(
    database.postDao().posts(),
    config)
  return livePageListBuilder
}

There are two changes to note here. First, you’re instantiating the RedditDb. Second, you’ve updated the LivePagedListBuilder to use the DataSource generated by Room instead of the one you built earlier.

Run the app. You should see an empty list with no content.

Paging Library Reddit Clone

Note: If you see any content in the list, please clear the data of the app or uninstall the app from your device and reinstall it again.

You’ve given Room the tools to generate a DataSource that pulls from the database. However, you don’t actually have anything in the database yet! How will you solve that problem?

Creating a BoundaryCallback

Here’s the flow you want to create:

  • First, load data into the Room database via the Reddit API.
  • You then want to use that data loaded into Room in the DataSource.Factory that Room will create for you to show posts in the list.
  • Once you get near the end of the available data in Room, you want to load more data into the database and start the process all over again.

However, there’s one problem: You don’t have a great way to know when you’re near the end of the available data in the DataSource. The Paging library provides a class dedicated to that purpose: BoundaryCallback.

BoundaryCallback signals when you should load new data. It lets you know when it has no data and when the item at the end of its available data has been loaded. Open the RedditBoundaryCallback class:

class RedditBoundaryCallback(private val db: RedditDb) :
    PagedList.BoundaryCallback<RedditPost>() {

  private val api = RedditService.createService()
  private val executor = Executors.newSingleThreadExecutor()
  private val helper = PagingRequestHelper(executor)

  override fun onZeroItemsLoaded() {
    super.onZeroItemsLoaded()
  }

  override fun onItemAtEndLoaded(itemAtEnd: RedditPost) {
    super.onItemAtEndLoaded(itemAtEnd)
  }
}

There’s a few values to note here. You’re passing in the RedditDb to insert items into the database, you’re creating a RedditService to load items from the network, and you’ve got an Executor to determine what thread to use when running the network and database tasks.

There’s one more value there: the PagingRequestHelper.

As you load things from the network into the database, you need to make sure that you’re not doubling up on requests when you reach the bottom of the available data.

Unfortunately, that is logic you theoretically have to manage yourself.

Fortunately, the Android team already built a class that manages that logic for you in the Android Architecture Components samples. The team is planning on moving that class into the Paging library, but it’s not quite there yet. So, for now, you should copy the PagingRequestHelper class into the project.

You can find a link to the original file here: PagingRequestHelper.

Fleshing Out the BoundaryCallback

First, you’re going to tackle the onZeroItemsLoaded() method. Add the following code belown super.onZeroItemsLoaded():

//1
helper.runIfNotRunning(PagingRequestHelper.RequestType.INITIAL) { helperCallback ->
  api.getPosts()
      //2
      .enqueue(object : Callback<RedditApiResponse> {

        override fun onFailure(call: Call<RedditApiResponse>?, t: Throwable) {
          //3
          Log.e("RedditBoundaryCallback", "Failed to load data!")
          helperCallback.recordFailure(t)
        }

        override fun onResponse(
            call: Call<RedditApiResponse>?,
            response: Response<RedditApiResponse>) {
          //4
          val posts = response.body()?.data?.children?.map { it.data }
          executor.execute {
            db.postDao().insert(posts ?: listOf())
            helperCallback.recordSuccess()
          }
        }
      })
}

This may seem like a lot of code, but it’s pretty simple when you break it down.

  1. You’re telling the PagingRequestHelper to run the following block if it’s not already running. Then, you let the PagingRequestHelper know that the block you’re trying to run is the INITIAL download. Finally, you are given a PagingRequestHelper.Request.Callback object to communicate with once you’re done. Pretty simple.
  2. This is a regular Retrofit call. You are asking the Reddit API to fetch some posts for you, and then you have callbacks to handle the success and failure blocks.
  3. If the API call fails, you log it to the console and let the helperCallback know that you have finished and that the call has failed. Letting the helper know will make future attempts able to run since the helper will not be waiting for you to finish the call.
  4. In the success block, You fetch the RedditPosts from the response you got back from the API. Then, you use the Executor that you declared above to execute a call to insert the new items into the database. Once that’s finished, you make sure to let the helper know that you have successfully inserted items into the database.

Not too bad! Time to move onto the next method: onItemAtEndLoaded(itemAtEnd:). For this method, you want to do almost the same thing. Add this code below super.onItemAtEndLoaded(itemAtEnd):

helper.runIfNotRunning(PagingRequestHelper.RequestType.AFTER) { helperCallback ->
  api.getPosts(after = itemAtEnd.key)
      .enqueue(object : Callback<RedditApiResponse> {

        override fun onFailure(call: Call<RedditApiResponse>?, t: Throwable) {
          Log.e("RedditBoundaryCallback", "Failed to load data!")
          helperCallback.recordFailure(t)
        }

        override fun onResponse(
            call: Call<RedditApiResponse>?,
            response: Response<RedditApiResponse>) {

          val posts = response.body()?.data?.children?.map { it.data }
          executor.execute {
            db.postDao().insert(posts ?: listOf())
            helperCallback.recordSuccess()
          }
        }
      })
}

The only differences here are that you’re passing the AFTER request type to the helper, and you’re using the key on the RedditPost object to fetch the next set of items. Each RedditPost object also has a key in addition to the key you get wrapping the whole list. If you didn’t have a key for each RedditPost, you would have to save what the last key was and look it up before you made the network call.

And that’s it! You’re done with the BoundaryCallback. Now, when the DataSource provided by Room runs out of data, you can use the BoundaryCallback to load more.

All that’s left is wiring everything!

Adding Final Touches

Open up the MainActivity class, again. Once again, you want to modify the initializedPagedListBuilder() method. Simply add this line of code before the return statement:

livePageListBuilder.setBoundaryCallback(RedditBoundaryCallback(database))

You’re now setting the BoundaryCallback on the LivePageListBuilder before returning it.

So that’s it! If you run the app, you should see data paged correctly. If you put your device into Airplane mode or turn off both Wifi and mobile data and restart the app, you should see the paging continues to work. Technically the app is now pulling data from the database instead of the network.

Where to Go From Here?

Remember, you can download the final project using the Download materials button at the top or bottom of this tutorial.

In this Paging library tutorial, you learned:

  • The different components of the Paging library.
  • The features and uses of DataSource.
  • How to build up a PagedListAdapter.
  • The best way to wire everything together with a LiveData.
  • How to add a Database layer for caching.

You’ve only scratched the surface of the Android Architecture Components and the Paging library. Check out the official documentation for more information: Android Developers | Library Overview

If you have any question or want to give me a thumbs up to start my comedic career, join the discussion below!

Contributors

Comments