johdougss
/
laravel-performance
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update massive

main
Johnathan Douglas 2023-07-07 18:37:17 -03:00
parent 71923923c3
commit 36577aa2e6
18 changed files with 633 additions and 77 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

76
README.md
View File

@ -1,66 +1,26 @@
<p align="center"><a href="https://laravel.com" target="_blank"><img src="https://raw.githubusercontent.com/laravel/art/master/logo-lockup/5%20SVG/2%20CMYK/1%20Full%20Color/laravel-logolockup-cmyk-red.svg" width="400" alt="Laravel Logo"></a></p>
# Install
<p align="center">
<a href="https://github.com/laravel/framework/actions"><img src="https://github.com/laravel/framework/workflows/tests/badge.svg" alt="Build Status"></a>
<a href="https://packagist.org/packages/laravel/framework"><img src="https://img.shields.io/packagist/dt/laravel/framework" alt="Total Downloads"></a>
<a href="https://packagist.org/packages/laravel/framework"><img src="https://img.shields.io/packagist/v/laravel/framework" alt="Latest Stable Version"></a>
<a href="https://packagist.org/packages/laravel/framework"><img src="https://img.shields.io/packagist/l/laravel/framework" alt="License"></a>
</p>
create database `example-update-massive`
## About Laravel
configure `.env`
Laravel is a web application framework with expressive, elegant syntax. We believe development must be an enjoyable and creative experience to be truly fulfilling. Laravel takes the pain out of development by easing common tasks used in many web projects, such as:
```dotenv
DB_CONNECTION=pgsql
DB_HOST=postgres
DB_PORT=5432
DB_DATABASE=example-update-massive
DB_USERNAME=default
DB_PASSWORD=secret
- [Simple, fast routing engine](https://laravel.com/docs/routing).
- [Powerful dependency injection container](https://laravel.com/docs/container).
- Multiple back-ends for [session](https://laravel.com/docs/session) and [cache](https://laravel.com/docs/cache) storage.
- Expressive, intuitive [database ORM](https://laravel.com/docs/eloquent).
- Database agnostic [schema migrations](https://laravel.com/docs/migrations).
- [Robust background job processing](https://laravel.com/docs/queues).
- [Real-time event broadcasting](https://laravel.com/docs/broadcasting).
QUEUE_CONNECTION=database
Laravel is accessible, powerful, and provides tools required for large, robust applications.
ITEMS_COUNT=10000
```
## Learning Laravel
`ITEMS_COUNT` é a variavel que irá definir quantos registros a fatory irá criar no banco de dados.
Laravel has the most extensive and thorough [documentation](https://laravel.com/docs) and video tutorial library of all modern web application frameworks, making it a breeze to get started with the framework.
run migrate;
You may also try the [Laravel Bootcamp](https://bootcamp.laravel.com), where you will be guided through building a modern Laravel application from scratch.
If you don't feel like reading, [Laracasts](https://laracasts.com) can help. Laracasts contains over 2000 video tutorials on a range of topics including Laravel, modern PHP, unit testing, and JavaScript. Boost your skills by digging into our comprehensive video library.
## Laravel Sponsors
We would like to extend our thanks to the following sponsors for funding Laravel development. If you are interested in becoming a sponsor, please visit the Laravel [Patreon page](https://patreon.com/taylorotwell).
### Premium Partners
- **[Vehikl](https://vehikl.com/)**
- **[Tighten Co.](https://tighten.co)**
- **[Kirschbaum Development Group](https://kirschbaumdevelopment.com)**
- **[64 Robots](https://64robots.com)**
- **[Cubet Techno Labs](https://cubettech.com)**
- **[Cyber-Duck](https://cyber-duck.co.uk)**
- **[Many](https://www.many.co.uk)**
- **[Webdock, Fast VPS Hosting](https://www.webdock.io/en)**
- **[DevSquad](https://devsquad.com)**
- **[Curotec](https://www.curotec.com/services/technologies/laravel/)**
- **[OP.GG](https://op.gg)**
- **[WebReinvent](https://webreinvent.com/?utm_source=laravel&utm_medium=github&utm_campaign=patreon-sponsors)**
- **[Lendio](https://lendio.com)**
## Contributing
Thank you for considering contributing to the Laravel framework! The contribution guide can be found in the [Laravel documentation](https://laravel.com/docs/contributions).
## Code of Conduct
In order to ensure that the Laravel community is welcoming to all, please review and abide by the [Code of Conduct](https://laravel.com/docs/contributions#code-of-conduct).
## Security Vulnerabilities
If you discover a security vulnerability within Laravel, please send an e-mail to Taylor Otwell via [taylor@laravel.com](mailto:taylor@laravel.com). All security vulnerabilities will be promptly addressed.
## License
The Laravel framework is open-sourced software licensed under the [MIT license](https://opensource.org/licenses/MIT).
```shell
php artisan migrate:fresh --seed
```

64
app/Console/Commands/UpdateMassiveCommand.php
View File

@ -2,8 +2,13 @@
namespace App\Console\Commands;
use App\Jobs\UpdateMassiveJob;
use App\Jobs\UpdateMassive1Job;
use App\Jobs\UpdateMassive2Job;
use App\Jobs\UpdateMassive3Job;
use App\Jobs\UpdateMassive4Job;
use App\Jobs\UpdateMassive5Job;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Bus;
class UpdateMassiveCommand extends Command
{
@ -26,6 +31,61 @@ class UpdateMassiveCommand extends Command
*/
public function handle()
{
UpdateMassiveJob::dispatch();
Bus::chain([
/**
* Atualiza o campo data e valor, uma transação por vez.
* Atualizando de forma unitária com where('id', 1), where('id', 2).
*
* 1000 - App\Jobs\UpdateMassive1Job .......... 1s DONE
* 10000 - App\Jobs\UpdateMassive1Job .......... 14s DONE
* 100000 - App\Jobs\UpdateMassive1Job ....... 2m 36s DONE
* 1000000 - App\Jobs\UpdateMassive1Job ...... 26m 10s DONE
*/
new UpdateMassive1Job(),
/**
* Atualiza o campo data e valor, mas passando todos os ids das transações,
* Atualizando com where id in(1,2,3...).
*
* 1000 - App\Jobs\UpdateMassive2Job ...... 19.87ms DONE
* 10000 - App\Jobs\UpdateMassive2Job ..... 149.19ms DONE
* 100000 - App\Jobs\UpdateMassive2Job ........... 1s DONE
* 1000000 - App\Jobs\UpdateMassive2Job .......... 15s DONE
*/
new UpdateMassive2Job(),
/**
* Atualiza o campo data e valor, mas o campo data será preenchido com um valor diferente para cada transação,
* Atualizando de forma unitária com where('id', 1), where('id', 2).
*
* 1000 - App\Jobs\UpdateMassive3Job ........... 1s DONE
* 10000 - App\Jobs\UpdateMassive3Job .......... 14s DONE
* 100000 - App\Jobs\UpdateMassive3Job ....... 2m 44s DONE
* 1000000 - App\Jobs\UpdateMassive3Job ...... 27m 10s DONE
*/
new UpdateMassive3Job(),
/**
* Atualiza o campo data e valor, mas o campo data será preenchido com um valor diferente para cada transação,
* Atualizando de forma multipla com UpdateMassive
*
* 1000 - App\Jobs\UpdateMassive4Job ..... 85.32ms DONE
* 10000 - App\Jobs\UpdateMassive4Job .... 737.00ms DONE
* 100000 - App\Jobs\UpdateMassive4Job .......... 8s DONE
* 1000000 - App\Jobs\UpdateMassive4Job ...... 1m 15s DONE
*/
new UpdateMassive4Job(),
/**
* Atualiza o campo data e valor, mas o campo data será preenchido com um valor diferente para cada transação,
* Atualizando de forma multipla com UpdateMassive2
*
* 1000 - App\Jobs\UpdateMassive5Job ..... 64.16ms DONE
* 10000 - App\Jobs\UpdateMassive5Job .... 590.20ms DONE
* 100000 - App\Jobs\UpdateMassive5Job .......... 7s DONE
* 1000000 - App\Jobs\UpdateMassive5Job ......... 58s DONE
*/
new UpdateMassive5Job(),
])->dispatch();
}
}

2
app/Console/Kernel.php
View File

@ -20,7 +20,7 @@ class Kernel extends ConsoleKernel
*/
protected function commands(): void
{
$this->load(__DIR__.'/Commands');
$this->load(__DIR__ . '/Commands');
require base_path('routes/console.php');
}

81
app/Database/UpdateMassive.php 100644
View File

@ -0,0 +1,81 @@
<?php
namespace App\Database;
use Illuminate\Database\DatabaseManager;
use Illuminate\Database\Query\Expression;
class UpdateMassive
{
/**
* UPDATE transactions
* SET updated_at = CASE id
* WHEN 1 THEN '2019-01-15 10:00:00'::date
* WHEN 2 THEN '2019-01-16 11:00:00'::date
* WHEN 3 THEN '2019-01-17 12:00:00'::date
* END
* WHERE id IN (1, 2, 3);
*
* update transactions set updated_at = '2019-01-15 10:00:00' where id = 1
* update transactions set updated_at = '2019-01-16 10:00:00' where id = 2
* update transactions set updated_at = '2019-01-17 10:00:00' where id = 3
*
* @param string $table
* @param string $primaryKeyName
* @param $config
*
* Config:
* [
* [
* "primary_key" => 200576
* "columns" => array:1 [
* "transactions_banks.coupon_code" => "2001462387"
* ]
* ]
* ]
*
* @return void
*/
public function apply(string $table, string $primaryKeyName, $config): void
{
/** @var DatabaseManager $db */
$db = app('db');
$primaryKeyValues = [];
$whenGroup = [];
foreach ($config as $configItem) {
$primaryKey = $configItem['primary_key'];
$primaryKeyValues[] = $primaryKey;
foreach ($configItem['columns'] as $columnName => $columnValue) {
if (!isset($whenGroup[$columnName])) {
$whenGroup[$columnName] = [];
}
$whenGroup[$columnName][] = vsprintf('WHEN %s THEN %s', [
$primaryKey,
$columnValue
]);
}
}
$values = [];
foreach ($whenGroup as $column => $whenArr) {
/** @var Expression[] $whenArr */
$case = $db->raw(vsprintf('CASE %s %s END', [$primaryKeyName, implode(' ', $whenArr)]));
$values[$column] = $case;
}
if (empty($values)) {
return;
}
// $db->enableQueryLog();
$db->table($table)
->whereIn($primaryKeyName, $primaryKeyValues)
->update($values);
// dd($db->getQueryLog());
}
}

101
app/Database/UpdateMassive2.php 100644
View File

@ -0,0 +1,101 @@
<?php
namespace App\Database;
use Illuminate\Support\Facades\DB;
class UpdateMassive2
{
/**
*
* update test as t set
* column_a = c.column_a,
* column_c = c.column_c
* from (values
* ('123', 1, '---'),
* ('345', 2, '+++')
* ) as c(column_b, column_a, column_c)
* where c.column_b = t.column_b;
*
*
* UPDATE transactions
* SET updated_at = CASE id
* WHEN 1 THEN '2019-01-15 10:00:00'::date
* WHEN 2 THEN '2019-01-16 11:00:00'::date
* WHEN 3 THEN '2019-01-17 12:00:00'::date
* END
* WHERE id IN (1, 2, 3);
*
* update transactions set updated_at = '2019-01-15 10:00:00' where id = 1
* update transactions set updated_at = '2019-01-16 10:00:00' where id = 2
* update transactions set updated_at = '2019-01-17 10:00:00' where id = 3
*
* @param string $table
* @param string $primaryKeyName
* @param $config
*
* Config:
* [
* [
* "primary_key" => 200576
* "columns" => array:1 [
* "transactions_banks.coupon_code" => "2001462387"
* ]
* ]
* ]
*
* @return void
*/
public function apply(string $table, string $primaryKeyName, array $config): void
{
/**
* update test as t set
* column_a = c.column_a,
* column_c = c.column_c
* from (values
* ('123', 1, '---'),
* ('345', 2, '+++')
* ) as c(column_b, column_a, column_c)
* where c.column_b = t.column_b;
*
*
* update transactions as t
* set date = c.date,
* value = c.value
* from (
* values (1, '2023-01-05 00:00:00'::timestamp, 23.44),
* (2, '2023-01-03 00:00:00'::timestamp, 23.44)
* ) as c(id, date, value)
* where t.id = c.id;
*
*/
$valuesSqls = [];
$columns = $config[0]['columns'];
foreach ($config as $configItem) {
$valuesSqls[] = vsprintf('(%s, %s)', [$configItem['primary_key'], implode(', ', $configItem['columns'])]);
}
$valuesSql = implode(', ', $valuesSqls);
$setSqls = [];
foreach ($columns as $columnName => $columnValue) {
$setSqls[] = vsprintf('%s = a2.%s', [$columnName, $columnName]);
}
$setSql = implode(', ', $setSqls);
$sql = vsprintf('update %s as a1 set %s from (values %s) as a2(%s, date, value) where a1.%s = a2.%s', [
$table,
$setSql,
$valuesSql,
$primaryKeyName,
$primaryKeyName,
$primaryKeyName,
]);
DB::statement($sql);
}
}

13
app/Jobs/UpdateMassiveJob.php → app/Jobs/UpdateMassive1Job.php
View File

@ -2,17 +2,24 @@
namespace App\Jobs;
use App\Tasks\UpdateMassiveTask;
use App\Tasks\UpdateMassive1Task;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
class UpdateMassiveJob implements ShouldQueue
class UpdateMassive1Job implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* The number of seconds the job can run before timing out.
*
* @var int
*/
public $timeout = 3600; //20min
/**
* Create a new job instance.
*/
@ -26,6 +33,6 @@ class UpdateMassiveJob implements ShouldQueue
*/
public function handle(): void
{
(new UpdateMassiveTask())->handle();
(new UpdateMassive1Task())->handle();
}
}

38
app/Jobs/UpdateMassive2Job.php 100644
View File

@ -0,0 +1,38 @@
<?php
namespace App\Jobs;
use App\Tasks\UpdateMassive2Task;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
class UpdateMassive2Job implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* The number of seconds the job can run before timing out.
*
* @var int
*/
public $timeout = 3600; //20min
/**
* Create a new job instance.
*/
public function __construct()
{
//
}
/**
* Execute the job.
*/
public function handle(): void
{
(new UpdateMassive2Task())->handle();
}
}

38
app/Jobs/UpdateMassive3Job.php 100644
View File

@ -0,0 +1,38 @@
<?php
namespace App\Jobs;
use App\Tasks\UpdateMassive3Task;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
class UpdateMassive3Job implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* The number of seconds the job can run before timing out.
*
* @var int
*/
public $timeout = 3600; //20min
/**
* Create a new job instance.
*/
public function __construct()
{
//
}
/**
* Execute the job.
*/
public function handle(): void
{
(new UpdateMassive3Task())->handle();
}
}

38
app/Jobs/UpdateMassive4Job.php 100644
View File

@ -0,0 +1,38 @@
<?php
namespace App\Jobs;
use App\Tasks\UpdateMassive4Task;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
class UpdateMassive4Job implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* The number of seconds the job can run before timing out.
*
* @var int
*/
public $timeout = 3600; //20min
/**
* Create a new job instance.
*/
public function __construct()
{
//
}
/**
* Execute the job.
*/
public function handle(): void
{
(new UpdateMassive4Task())->handle();
}
}

38
app/Jobs/UpdateMassive5Job.php 100644
View File

@ -0,0 +1,38 @@
<?php
namespace App\Jobs;
use App\Tasks\UpdateMassive5Task;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
class UpdateMassive5Job implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* The number of seconds the job can run before timing out.
*
* @var int
*/
public $timeout = 3600; //20min
/**
* Create a new job instance.
*/
public function __construct()
{
//
}
/**
* Execute the job.
*/
public function handle(): void
{
(new UpdateMassive5Task())->handle();
}
}

2
app/Models/Transaction.php
View File

@ -9,6 +9,8 @@ class Transaction extends Model
{
use HasFactory;
protected $table = 'transactions';
/**
* The attributes that are mass assignable.
*

11
app/Tasks/UpdateMassiveTask.php → app/Tasks/UpdateMassive1Task.php
View File

@ -5,17 +5,17 @@ namespace App\Tasks;
use App\Models\Transaction;
use Carbon\Carbon;
class UpdateMassiveTask
class UpdateMassive1Task
{
public function handle()
{
echo 'update massive unit' . PHP_EOL;
Transaction::query()
->select([
'id',
])
->where('id', '>', 0)
->chunkById(10, function ($transactions) {
->chunkById(1000, function ($transactions) {
foreach ($transactions as $transaction) {
Transaction::query()
->where('id', '=', $transaction->id)
@ -25,5 +25,6 @@ class UpdateMassiveTask
]);
}
});
}
}

27
app/Tasks/UpdateMassive2Task.php 100644
View File

@ -0,0 +1,27 @@
<?php
namespace App\Tasks;
use App\Models\Transaction;
use Carbon\Carbon;
class UpdateMassive2Task
{
public function handle()
{
Transaction::query()
->select([
'id',
])
->where('id', '>', 0)
->chunkById(1000, function ($transactions) {
Transaction::query()
->whereIn('id', $transactions->pluck('id'))
->update([
'date' => Carbon::now(),
'value' => 1,
]);
});
}
}

35
app/Tasks/UpdateMassive3Task.php 100644
View File

@ -0,0 +1,35 @@
<?php
namespace App\Tasks;
use App\Models\Transaction;
use Carbon\Carbon;
class UpdateMassive3Task
{
public function handle(): void
{
Transaction::query()
->select([
'id',
'date',
'value',
])
->where('id', '>', 0)
->chunkById(1000, function ($transactions) {
foreach ($transactions as $transaction) {
$transaction->date = Carbon::now()->subDays(random_int(1, 3));
$transaction->value = 10 + random_int(0, 10);
Transaction::query()
->where('id', '=', $transaction->id)
->update([
'date' => $transaction->date,
'value' => $transaction->value,
]);
}
});
}
}

42
app/Tasks/UpdateMassive4Task.php 100644
View File

@ -0,0 +1,42 @@
<?php
namespace App\Tasks;
use App\Database\UpdateMassive;
use App\Models\Transaction;
use Carbon\Carbon;
use Ramsey\Collection\Collection;
class UpdateMassive4Task
{
public function handle(): void
{
Transaction::query()
->select([
'id',
'date',
'value',
])
->where('id', '>', 0)
->chunkById(1000, function ($transactions) {
$config = [];
/** @var Collection $transactions */
foreach ($transactions as $transaction) {
$transaction->date = Carbon::now()->subDays(random_int(1, 3));
$transaction->value = 10 + random_int(0, 10);
$config[] = [
'primary_key' => $transaction->id,
'columns' => [
'transactions.date' => "'$transaction->date'::timestamp",
'transactions.value' => $transaction->value,
]
];
}
$updateMassive = new UpdateMassive();
$updateMassive->apply('transactions', 'id', $config);
});
}
}

41
app/Tasks/UpdateMassive5Task.php 100644
View File

@ -0,0 +1,41 @@
<?php
namespace App\Tasks;
use App\Database\UpdateMassive2;
use App\Models\Transaction;
use Carbon\Carbon;
use Illuminate\Support\Collection;
class UpdateMassive5Task
{
public function handle(): void
{
Transaction::query()
->select([
'id',
'date',
'value',
])
->where('id', '>', 0)
->chunkById(1000, function ($transactions) {
/** @var Collection $transactions */
foreach ($transactions as $transaction) {
$transaction->date = Carbon::now()->subDays(random_int(1, 3));
$transaction->value = 10 + random_int(0, 10);
$config[] = [
'primary_key' => $transaction->id,
'columns' => [
'date' => "'$transaction->date'::timestamp",
'value' => $transaction->value,
]
];
}
$updateMassive2 = new UpdateMassive2();
$updateMassive2->apply('transactions', 'id', $config);
});
}
}

20
config/update-massive.php 100644
View File

@ -0,0 +1,20 @@
<?php
return [
/*
|--------------------------------------------------------------------------
| Default Session Driver
|--------------------------------------------------------------------------
|
| This option controls the default session "driver" that will be used on
| requests. By default, we will use the lightweight native driver but
| you may specify any of the other wonderful drivers provided here.
|
| Supported: "file", "cookie", "database", "apc",
| "memcached", "redis", "dynamodb", "array"
|
*/
'items_count' => env('ITEMS_COUNT', 1000)
];

43
database/seeders/TransactionSeeder.php
View File

@ -2,7 +2,6 @@
namespace Database\Seeders;
// use Illuminate\Database\Console\Seeds\WithoutModelEvents;
use App\Models\Transaction;
use Illuminate\Database\Seeder;
@ -13,28 +12,56 @@ class TransactionSeeder extends Seeder
*/
public function run(): void
{
$count = 100;
$type = 2;
$count = config('update-massive.items_count');
$type = 3;
if ($type === 1) {
/**
* insert unit
* 1000 - Database\Seeders\TransactionSeeder ...... 1,443.86 ms DONE
* 10000 - Database\Seeders\TransactionSeeder ..... 14,256.24 ms DONE
* 1000 - Database\Seeders\TransactionSeeder ...... 1,443.86 ms DONE
* 10000 - Database\Seeders\TransactionSeeder ..... 14,256.24 ms DONE
* 100000 - Database\Seeders\TransactionSeeder .... 158,413.75 ms DONE
*/
$transactions = Transaction::factory($count)->create();
Transaction::factory($count)->create();
}
if ($type === 2) {
/**
* insert multiple
* 1000 - Database\Seeders\TransactionSeeder ......... 96.11 ms DONE
* 10000 - Database\Seeders\TransactionSeeder ........ 887.48 ms DONE
* 1000 - Database\Seeders\TransactionSeeder ......... 96.11 ms DONE
* 10000 - Database\Seeders\TransactionSeeder ........ 887.48 ms DONE
* 100000 - SQLSTATE[HY000]: General error: 7 number of parameters must be between 0 and 65535
*/
$transactions = Transaction::factory($count)->make();
Transaction::query()->insert($transactions->toArray());
}
if ($type === 3) {
/**
* Limit insert sql string
* insert multiple block
*
* 1000 - Database\Seeders\TransactionSeeder ......... 97.71 ms DONE
* 10000 - Database\Seeders\TransactionSeeder ........ 833.27 ms DONE
* 100000 - Database\Seeders\TransactionSeeder ...... 8,256.29 ms DONE
* 100000 - Database\Seeders\TransactionSeeder .... 95,473.46 ms DONE
*/
$block = 1000;
while ($count > 0) {
if ($count < $block) {
$block = $count;
}
$transactions = Transaction::factory($block)->make();
Transaction::query()->insert($transactions->toArray());
$count -= $block;
}
}
}
}