Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

Is there "S3 range read function" that allows to read assigned byte range from AWS-S3 file?

Ask Question

Trying to process large file in AWS Lamba and skipping through the whole file seems a bit wasteful. Is there a "range read" function that allows to read only predefined byte range from S3 file?

Yes, this is possible. According to S3 documentation of GET Object in the REST API, it supports use of the HTTP Range header.

Range

Downloads the specified range bytes of an object. For more information about the HTTP Range header, go to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 .

In the example syntax:

GET /ObjectName HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Authorization: authorization string (see Authenticating Requests (AWS Signature Version 4))
Range:bytes=byte_range
Popular S3 client libraries, such as the AWS SDK for Java provide convenient client-side APIs for specifying the range information.
                @olekb , I'm not certain when it was added, but I know it was usable at least since early 2016.  My experience is using it in Apache Hadoop in the S3AInputStream class.
– Chris Nauroth
                Jul 14, 2017 at 17:45
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.