Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

529
Views
TRIM_HORIZON vs LATEST

I can't find in the formal documentation of AWS Kinesis any explicit reference between TRIM_HORIZON and the checkpoint, and also any reference between LATEST and the checkpoint.

Can you confirm my theory:

  • TRIM_HORIZON - In case the application-name is new, then I will read all the records available in the stream. Else, application-name was already used, then I will read from my last checkpoint.

  • LATEST - In case the application-name is new, then I will read all the records in the stream which added after I subscribed to the stream. Else, application-name was already used, I will read messages from my last checkpoint.

  • The difference between TRIM_HORIZON and LATEST is only in case the application-name is new.

over 4 years ago · Santiago Trujillo
3 answers
Answer question

0

AT_TIMESTAMP

-- from specific time stamp

TRIM_HORIZON

-- all the available messages in Kinesis stream from the beginning (same as earliest in Kafka)

LATEST

-- from the latest messages , i.e current message that just came into Kinesis/Kafka and all the incoming messages from that time onwords

over 4 years ago · Santiago Trujillo Report

0

From GetShardIterator documentation (which lines up with my experience using Kinesis):

In the request, you can specify the shard iterator type AT_TIMESTAMP to read records from an arbitrary point in time, TRIM_HORIZON to cause ShardIterator to point to the last untrimmed record in the shard in the system (the oldest data record in the shard), or LATEST so that you always read the most recent data in the shard.

Basically, the difference is whether you want to start from the oldest record (TRIM_HORIZON), or from "right now" (LATEST - skipping data between latest checkpoint and now).

over 4 years ago · Santiago Trujillo Report

0

The question clearly asks how these options relate to the checkpoint. However, none of the existing answers addresses the checkpoint at all.

An authoritative answer to this question by Justin Pfifer appears in a GitHub issue here.

The most relevant portion is

The KCL will always use the value in the lease table if it's present. It's important to remember that Kinesis itself doesn't track the position of consumers. Tracking is provided by the lease table. Leases in the KCL server double duty. They provide both mutual exclusion, and position tracking. So for mutual exclusion a lease needs to be created, and to satisfy the position tracking an initial value must be selected.

(Emphasis added by me.)

over 4 years ago · Santiago Trujillo Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!