This weblog article has been cross-posted from the Delta.io weblog.

We’re excited for the launch of Delta Sharing 0.4.0 for the open-source knowledge lake challenge Delta Lake. The newest launch introduces a number of key enhancements and bug fixes, together with the next options:

• Delta Sharing is now out there for Google Cloud Storage – Now you can share Delta Tables on the Google Cloud Platform (#81, #105)
• A brand new API for getting the metadata of a Delta Share – a brand new GetShare REST API has been added for querying a Share by its title (#95, #97)
• Delta Sharing Protocol and REST API enhancements – the Delta Sharing protocol has been prolonged to incorporate the Share Id and Desk Ids, as effectively improved response codes and error codes (#85, #89, #93, #98)
• Customise a recipient sharing profile within the Apache Spark™ connector – a brand new Delta Sharing Profile Supplier has been added to the Spark connector to allow simpler entry of the sharing profile (#99, #107)

On this weblog submit, we are going to undergo every of the enhancements on this launch.

Delta Sharing on Google Cloud Storage

New to this launch, now you can share Delta Tables in Google Cloud Storage utilizing the reference implementation of a Delta Sharing Server.

Delta Sharing on Google Cloud Storage instance

Sharing Delta Tables on Google Cloud Storage is less complicated than ever! For instance, to share a Delta Desk known as “time”, you possibly can merely replace the Delta Sharing server configuration with the situation of the Delta desk on Google Cloud Storage:

model: 1
shares:
- title: "vaccineshare"
 schemas:
 - title: "samplecoviddata"
   tables:
   - title: "time"
     location: "gs://deltasharingexample/COVID/Time"

Delta Sharing Server configuration file containing the situation to a Delta desk on Google Cloud Storage.
The Delta Sharing server will routinely course of the information on Google Cloud Storage for a Delta Sharing question.
Authenticating with Google Cloud Storage
The Delta Sharing Server acts as a gatekeeper to the underlying knowledge in a Delta Share. When a recipient queries a Delta desk in a Delta Share, the Delta Sharing Server first checks the permissions to ensure the information recipient has entry to knowledge. Subsequent, if entry is permitted, the Delta Sharing Server will have a look at the file objects that make up the Delta desk and well filter down the recordsdata if a predicate is included within the question, for instance. Lastly, the Delta Sharing Server will generate short-lived, pre-signed URLs that enable the information recipient to entry the recordsdata, or subset of recordsdata, from the Delta Sharing Consumer straight from cloud storage somewhat than streaming the information by the Delta Sharing Server.

With a view to generate the short-lived file URLs, the Delta Sharing Server makes use of a Service Account to learn Delta tables from Google Cloud Storage. To configure the Service Account credentials, you possibly can set the surroundings variable GOOGLE_APPLICATION_CREDENTIALS earlier than beginning the Delta Sharing Server.
# Delta Sharing Server Setting Variable

export GOOGLE_APPLICATION_CREDENTIALS="/config/keyfile.json"

New API for getting a Delta Share
Generally, it could be useful for a recipient to test in the event that they nonetheless have entry to a Delta Share. This launch provides a brand new REST API, GetShare, in order that customers can rapidly take a look at if a Delta Share has exceeded its expiration time.
For instance, to test in case you nonetheless have entry to a Delta Share you possibly can merely ship a GET request to the /shares/{share_name} endpoint on the sharing server:
import requests
import json

response = requests.get(
   "http://localhost:8080/delta-sharing/shares/airports",
   headers={
       "Authorization":"Bearer token"
   }
)
print(json.dumps(response.json(), indent=2))

Instance GET request despatched to the sharing server that permits recipients to test whether or not or not they nonetheless have entry to a Delta Share.
{
   "share": {
       "title": "airports"
   }
}

Instance response obtained from the GetShare REST API that’s new to the Delta Sharing 0.4.0 launch.
If the Delta Share has exceeded its expiration, the Sharing server will reply with a 403 HTTP error code.
Delta Sharing protocol enhancements
Included on this launch are improved error codes and error messages within the Delta Sharing protocol definition. For instance, if a Delta Share is just not situated on the Delta Sharing Server, an error code and error message containing the main points of the error is now included on this launch.
import requests
import json
 
response = requests.get(
   "http://localhost:8080/delta-sharing/shares/yellowcab",
   headers={
       "Authorization":"Bearer token"
   }
)
print(json.dumps(response.json(), indent=2))

Instance GET request for a Share that doesn’t exist on the Delta Sharing Server.
{
   "errorCode": "RESOURCE_DOES_NOT_EXIST",
   "message": "share 'yellowcab' not discovered"
}

Instance response containing an improved error code and particulars in regards to the error that’s new to the Delta Sharing 0.4.0 launch.
Moreover, this launch extends the Delta Sharing Protocol to reply with the distinctive Delta Share and Desk Ids. Distinctive Ids assist the information recipient disambiguate the title of datasets as time passes. That is particularly helpful when the information recipient is a big group and desires to use entry management on the shared dataset inside their group
Customizing a recipient Sharing profile
The Delta Sharing profile file is a JSON configuration file that accommodates the data for a recipient to entry shared knowledge on a Delta Sharing server. A brand new supplier has been added on this launch that permits simpler entry to the Delta Sharing profile for knowledge recipients.
/**
 * A supplier that gives a Delta Sharing profile for knowledge 
 * recipients to entry the shared knowledge. 
 */
trait DeltaSharingProfileProvider {
 def getProfile: DeltaSharingProfile
}

The Delta Sharing profile file is a JSON configuration file that accommodates the data for a recipient to entry shared knowledge on a Delta Sharing server.
What’s subsequent
We’re already gearing up for a lot of new options within the subsequent launch of Delta Sharing. You possibly can observe all of the upcoming releases and deliberate options in GitHub milestones.

Credit
We’d like to increase a particular thanks for the contributions to this launch to Denny Lee, Lin Zhou, Shixiong Zhu, William Chau, Xiaotong Solar, Kohei Toshimitsu.