Ausnahme gefangen: SSL certificate problem: certificate is not yet valid ๐Ÿ“Œ How to Downsample Your Data Efficiently๏ผŸ

๐Ÿ  Team IT Security News

TSecurity.de ist eine Online-Plattform, die sich auf die Bereitstellung von Informationen,alle 15 Minuten neuste Nachrichten, Bildungsressourcen und Dienstleistungen rund um das Thema IT-Sicherheit spezialisiert hat.
Ob es sich um aktuelle Nachrichten, Fachartikel, Blogbeitrรคge, Webinare, Tutorials, oder Tipps & Tricks handelt, TSecurity.de bietet seinen Nutzern einen umfassenden รœberblick รผber die wichtigsten Aspekte der IT-Sicherheit in einer sich stรคndig verรคndernden digitalen Welt.

16.12.2023 - TIP: Wer den Cookie Consent Banner akzeptiert, kann z.B. von Englisch nach Deutsch รผbersetzen, erst Englisch auswรคhlen dann wieder Deutsch!

Google Android Playstore Download Button fรผr Team IT Security



๐Ÿ“š How to Downsample Your Data Efficiently๏ผŸ


๐Ÿ’ก Newskategorie: Programmierung
๐Ÿ”— Quelle: dev.to

Tired of spending countless mintues downsampling your data? Look no further!

In this article, youโ€™ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million minute-level records in only 41 seconds in DolphinDB.

The basic configuration of the DolphinDB server is:

16 CPU cores
256 GB memory
4 SSDs
A DolphinDB cluster with 4 data nodes is deployed, and each node uses a SSD.

Image description

The data we use is:

the level 1 quotes on August, 2007 from New York Stock Exchange
around 272 GB, with 6.48 billion records

Downsampling can be performed with a SQL statement in DolphinDB.

Image description

As the SQL query may involve multiple partitions, DolphinDB breaks down the job into several tasks and assigns the tasks to the corresponding data nodes for parallel execution. When all the tasks are completed, the system merges the intermediate results from the nodes to return the final result.

The script is as follows:



db = database("dfs://TAQ")
quotes = db.loadTable("quotes")
select count(*)  from quotes where date between 2007.08.01 : 2007.08.31

model=select  top 1 symbol,date, minute(time) as minute, bid, ofr from quotes where date=2007.08.01,symbol=`EBAY
if(existsTable("dfs://TAQ", "quotes_minute_sql"))
 db.dropTable("quotes_minute_sql")
db.createPartitionedTable(model, "quotes_minute_sql", `date`symbol)

timer{
 minuteQuotes=select avg(bid) as bid, avg(ofr) as ofr from quotes where data between 2007.08.01 : 2007.08.31 group by symbol,date,bar(time, 60) as minute
 loadTable("dfs://TAQ", "quotes_minute_sql").append!(minuteQuotes)
}

select count(*)  from loadTable("dfs://TAQ", "quotes_minute")

The frequency can be adjusted as needed just by modifying bar(time, 60). Here 60 means the data is downsampled to 1-minute interval as the timestamp values have seconds precision.

Image description

The table โ€œquotes_minute_sqlโ€œ is created with createPartitionedTable and the downsampled result can be appended to this table.

Image description

We can execute the script and visit the web-based user interface to check the resource usage. Itโ€™s shown that all CPU cores have participated in the downsampling. On each data node, 15 tasks are running concurrently as data is being read from disk.

Image description

When we come back to VScode and check the execution status, we find that it only takes 41 seconds to complete the data downsampling, which generates 61 million minute-level records.

Image description

DolphinDB exhibits outstanding performance in data downsampling due to the following reasons:

Jobs are executed distributedly and resources of different nodes can be utilized at the same time;
Compression reduces the disk I/O;
Columnar storage and vectorized computation improve the efficiency of aggregation.

To learn detailed operations of data downsampling, take a look at this demo!
https://youtu.be/0vRuiz1Lf6Y

Thanks for your reading! To keep up with our latest news, please follow our Twitter @DolphinDB_Inc, Linkedin and YouTube. You can also join our Slack to chat with the author!

...



๐Ÿ“Œ How to Downsample Your Data Efficiently๏ผŸ


๐Ÿ“ˆ 63.48 Punkte

๐Ÿ“Œ r0ehre: How to efficiently build cross-platform apps that your users love?


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ r0ehre: How to efficiently build cross-platform apps that your users love?


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Donโ€™t Dwell On It: How to Detect a Breach on Your Network More Efficiently


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Running tests in your games more efficiently, feat. SuperScale & Traplight (Playtime 2019 - Games)


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ IOTransfer 4: Manage Your iPhone/iPad Efficiently To Make Most Of It


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Efficiently use your Requisitions to upgrade in Star Wars: Squadrons


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Efficiently manage your cloudย operationsย using Cloud Adoption Framework


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ How To Efficiently Clean Up And Revive Your Mac


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Efficiently Manage Your GraphQL API with API Gateway


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Beyond Changing Technology: Scaling Your Applications Efficiently


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ How to Use Your Bug Bounty Budget Efficiently


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Test your Code Efficiently using pytest Module


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Efficiently manage your cloudย operationsย using Cloud Adoption Frameworkย ย  | Azure Enablement


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ App Tamer 2.7.6 - Efficiently manage your CPU cycles, save energy, and more.


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ Docusaurus: Create your project documentation simply and efficiently!


๐Ÿ“ˆ 21.18 Punkte

๐Ÿ“Œ How to efficiently manage PC refresh and data migration for a remote workforce


๐Ÿ“ˆ 21.02 Punkte

๐Ÿ“Œ Efficiently Managing Data with SELECT During INSERT Queries


๐Ÿ“ˆ 21.02 Punkte

๐Ÿ“Œ A simple way to efficiently calculate and represent average data inย Rails


๐Ÿ“ˆ 21.02 Punkte

๐Ÿ“Œ Docker Trick: Running Containers Efficiently Using A Bash Script


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ I have 3 hard drives in my computer, I want to install a Linux Distro onto my computer and have it efficiently use all 3 (Details in post)


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ How Corporations Efficiently Manage Mobile Devices?


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ Zsync HTTP-based File Transfer Utility Transfers Large Files Efficiently


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ Handling Device Orientation Efficiently in Vulkan With Pre-Rotation


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ Fixing Vulnerabilities Effectively & Efficiently - John Matherly - ASW #116


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ How to launch apps on Gnome Shell more efficiently


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ How to launch apps on Gnome Shell more efficiently


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ Businesses are failing to identify bugs in code efficiently


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ Organizations struggling to remedy cloud misfiguration issues efficiently


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ Vivaldi 1.7 Is Just Around the Corner, Let You Share Things More Efficiently


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ Reddit Search: How to Use Reddit Efficiently and Fast


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ Most federal government bodies in Brazil fail to use AI efficiently


๐Ÿ“ˆ 17.67 Punkte

๐Ÿ“Œ How MacOS Finder Quick Actions can help you work more efficiently


๐Ÿ“ˆ 17.67 Punkte











matomo