An Integrated Data Processing Framework for Enhancing Pretraining Data Quality of Foundation Models
An integrated data processing framework that automates data cleaning, deduplication, and quality evaluation to enhance the pretraining data for foundation models.
Open-Source ETL Pipeline for Efficiently Processing Large Language Model Data at Scale
Dataverse is an open-source, user-friendly ETL (Extract, Transform, Load) pipeline designed to efficiently process and analyze massive datasets for large language model development.
ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks
ShuffleBench introduces a new benchmark for evaluating stream processing frameworks' performance in large-scale data shuffling operations.
WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset
WanJuan-CC is a safe and high-quality English webtext dataset derived from Common Crawl, created through a meticulous process to ensure data safety and quality.
DynaWarp – Efficient, large-scale log storage and retrieval: Novel Indexing Structure for Real-time Data Processing
Efficient indexing structure DynaWarp offers significant storage savings and faster query throughput for large-scale log data processing.
© 2024 by Linnk AI