快速将大脚本分拆成多份并发执行脚本 巧用命令 更快更爽

 Yuema约吗?一起学技术,一起成长!学海无涯 高人带路系列

程序的世界,就是有坑的地方!分享踩坑的心得与体验!每天分享一点点!

关注公众号,进入学海无涯,高人带路模式!!​split​再难,有人带路,轻松搞定

命令比较晦涩

split -l 2 1.txt -d -a 4 data_&&ls|grep data_|xargs -n1 -i{} mv {} {}.sh

$ split --helpUsage: split [OPTION]... [FILE [PREFIX]]Output pieces of FILE to PREFIXaa, PREFIXab, ...;default size is 1000 lines, and default PREFIX is 'x'.With no FILE, or when FILE is -, read standard input.Mandatory arguments to long options are mandatory for short options too.  -a, --suffix-length=N   generate suffixes of length N (default 2)      --additional-suffix=SUFFIX  append an additional SUFFIX to file names  -b, --bytes=SIZE        put SIZE bytes per output file  -C, --line-bytes=SIZE   put at most SIZE bytes of records per output file  -d                      use numeric suffixes starting at 0, not alphabetic      --numeric-suffixes[=FROM]  same as -d, but allow setting the start value  -e, --elide-empty-files  do not generate empty output files with '-n'      --filter=COMMAND    write to shell COMMAND; file name is $FILE  -l, --lines=NUMBER      put NUMBER lines/records per output file  -n, --number=CHUNKS     generate CHUNKS output files; see explanation below  -t, --separator=SEP     use SEP instead of newline as the record separator;                            '\0' (zero) specifies the NUL character  -u, --unbuffered        immediately copy input to output with '-n r/...'      --verbose           print a diagnostic just before each                            output file is opened      --help     display this help and exit      --version  output version information and exitThe SIZE argument is an integer and optional unit (example: 10K is 10*1024).Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000).CHUNKS may be:  N       split into N files based on size of input  K/N     output Kth of N to stdout  l/N     split into N files without splitting lines/records  l/K/N   output Kth of N to stdout without splitting lines/records  r/N     like 'l' but use round robin distribution  r/K/N   likewise but only output Kth of N to stdoutGNU coreutils online help: <http://www.gnu.org/software/coreutils/>Report split translation bugs to <http://translationproject.org/team/>Full documentation at: <http://www.gnu.org/software/coreutils/split>or available locally via: info '(coreutils) split invocation'
不要造低级的轮子,这个轮子一点都不低级,如果自己去写代码的话,也好实现,不过,有现成的工具就用上吧,而且直接使用git bash就能完成脚本拆分。
这已经实现了昨天所期望的自动拆分脚本,而脚本生成,执行进度监控并没有实现,不过,感觉已经比较爽了。一个适度的平衡点吧!split -l 2 1.txt -d -a 4 data_&&ls|grep data_|xargs -n1 -i{} mv {} {}.sh 这轮子顶很多很多行代码了,如果写个程序进行大文本拆成多份文本,想想得多少行,这么一行命令就按定

作者:钟代麒

出处:http://www.jishudao.com/
版权归作者所有,转载请注明出处

巧用正则+Chrome彩蛋功能=快速生成cUrl脚本 fetch、cUrl cmd、cUrl bash、PowerShell

 Yuema约吗?一起学技术,一起成长!学海无涯 高人带路系列

程序的世界,就是有坑的地方!分享踩坑的心得与体验!每天分享一点点!
关注公众号,进入学海无涯,高人带路模式!!学会一招神技能​

今天遇到一个需要批量调接口的场景,从数据库中查出上万条数据,再按接口规则生成带get参数的url请求,还要带登陆状态。

一、从数据库中捞出数据

将查出来的数据贴到notepad++中,大概是这样子的数据,示范一下

1  a  112  b  223  c  334  d  445  e  556  f  667  g  778  h  889  i  99

将空格换成%20,这个直接把空格换掉就是,示例数据没有演示这个场景,自行脑补一下。

二、使用chrome生成一个带cookie的请求

操作截图如上图所示,是不是很方便,cookies都带上了,有登陆状态,调接口相当方面。

curl "http://yue.ma/?i=9&m=6&p=11" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecure

三、使用正则替换,生成批量脚本

制作一个脚本正则模板

curl "http://yue.ma/?i=$1&m=$2&p=$3" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecure

使用notepad++替换功能

查找目标填写:

^([^\t]+)\t([^\t]+)\t([^\t]+)$

替换为填写:

curl "http://yue.ma/?i=$1&m=$2&p=$3" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecure

生成脚本

curl "http://yue.ma/?i=1&m=a&p=11" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecurecurl "http://yue.ma/?i=2&m=b&p=22" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecurecurl "http://yue.ma/?i=3&m=c&p=33" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecurecurl "http://yue.ma/?i=4&m=d&p=44" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecurecurl "http://yue.ma/?i=5&m=e&p=55" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecurecurl "http://yue.ma/?i=6&m=f&p=66" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecurecurl "http://yue.ma/?i=7&m=g&p=77" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecurecurl "http://yue.ma/?i=8&m=h&p=88" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecurecurl "http://yue.ma/?i=9&m=i&p=99" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/78.0.3904.108 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: zh-CN,zh;q=0.9" -H "Cookie: __51cke__=; __tins__18251311=^%^7B^%^22sid^%^22^%^3A^%^201575548726427^%^2C^%^20^%^22vd^%^22^%^3A^%^203^%^2C^%^20^%^22expires^%^22^%^3A^%^201575551216664^%^7D; __51laig__=3" --compressed --insecure
执行脚本,我使用git bash执行的,直接新建一个bash.sh,把生成的代码放进去,执行脚本
./bash.sh
等着几万条调用执行吧!no no no,数据过多,可以拆成多份bash脚本执行,可以加快速度。这种执行方式有一个好处就是,不需要额外去写程序,直接一顿replace生成脚本跑起来。不好的就是命令太多的话,执行并没有进度显示。
可以写个辅助的脚本执行工具,自动拆份脚本并行执行,显示显示进度什么的。嗯嗯,学会一下^([^\t]+)\t([^\t]+)\t([^\t]+)$ 与chrome的脚本生成功能,非常不错!

作者:钟代麒

出处:http://www.jishudao.com/
版权归作者所有,转载请注明出处

FileBeat CPU占用率过高Case处理 | max_procs: 1

 Yuema约吗?一起学技术,一起成长!学海无涯 高人带路系列

程序的世界,就是有坑的地方!分享踩坑的心得与体验!每天分享一点点!
关注公众号,进入学海无涯,高人带路模式!!Filebeat​再难,有人带路,轻松搞定

Case: Filebeat内存占用率过高
处理​方案
max_procs: 1
问题渐进明细,遇到这种别人的东西CPU占用率过高,是不太好下手的​。解决这个问题的思路有2个,1是网上找踩过坑的资料,2是再现一下问题,寻找​思路。
一、找资料
FileBeat出现cpu占用过高,有说是FileBeat版本与kafka版本不一致,也有说监控文件过多,没有及时inactive的,还有就是限制c​pu使用数量的。最开始找资料的时候,并没有太看到限制cpu核数的​。所以​踩坑还是一踩到底。
二、再现问题
2.1 LogAuto模拟大量日志
为了重现问题,在本地启动一个Filebeat,输出为控制台,采集目录有多个,写了一个日志生成小工具。
using System;using System.Collections.Generic;using System.Configuration;using System.IO;using System.Linq;using System.Text;using System.Threading;namespace LogAuto{    class Program    {        static void Main(string[] args)        {            String siteName = ConfigurationManager.AppSettings["siteName"]??"default Site";            // 模拟 100个站点            List<LogThread> logThreadPool = new List<LogThread>();            for(int i=0;i<100;i++){                LogThread logThread = new LogThread();                logThread.logTag = String.Format("{0}", i);                logThread.siteName = siteName;                logThreadPool.Add(logThread);                new Thread(new ThreadStart(logThread.WriteLog)).Start();            }        }    }}
using System;using System.Collections.Generic;using System.IO;using System.Linq;using System.Text;namespace LogAuto{    class LogThread    {        public String logTag { get;set;}        public String siteName { get;set;}        public void WriteLog() {            while (true)            {                File.AppendAllText(String.Format("{0}.log.log", logTag), String.Format("产生一条日志 {0} {1}\r\n", siteName, DateTime.Now));                Console.WriteLine(String.Format("{2}输出一条日志:{0} {1}", siteName, DateTime.Now, logTag));                //Thread.Sleep(1000);            }        }    }}

​不管启多少个LogAuto,生成日志,filebeat采集日志,完全看不出Filebeat有什么不爽,于是,​这个模拟算是失败了。连多行日志都没有模拟出来,也没有让Filebeat​有任何压力。

​2.2 模拟大量线上日志

using System;using System.Collections.Generic;using System.Configuration;using System.Linq;using System.Net;using System.Text;using System.Threading;namespace LoadTest{    class Program    {        static void Main(string[] args)        {            int threadCount = int.Parse(ConfigurationManager.AppSettings["ThreadCount"] ?? "1000");            int threadSleep = int.Parse(ConfigurationManager.AppSettings["ThreadSleep"] ?? "1000");            String targetUrl = ConfigurationManager.AppSettings["TargetUrl"] ?? "http://yue.ma/?fbtest";            List<Thread> logThreadPool = new List<Thread>();            for (int i = 0; i < threadCount; i++)            {                Thread tread = new Thread((m) =>                {                    while (true) {                        try                        {                            WebClient wc = new WebClient();                            String data = wc.DownloadString(String.Format("{0}?r={1}", targetUrl, DateTime.Now));                            Console.WriteLine(String.Format("线程 {0} 返回结果 {1} 时间 {2}", m, data, DateTime.Now));                            Thread.Sleep(threadSleep);                        }                        catch (Exception ex) {                            Console.WriteLine(ex.Message);                        }                    }                });                logThreadPool.Add(tread);                tread.Start(i);            }            Console.WriteLine("输入任意字符退出");            Console.ReadKey();        }    }}

​启动了多个LoadTest实例,线程数都1000,压力是上去了​。发现FileBeat没有任何压力,完全不对​劲。查看FileBeat日志,filebeat压根就没有工作,kafka\logstash\elasticsearch统统没有工作。

​ 2.3 完整的日志收集环境+模拟大量线上日志

​这里要推荐linux下实用的history命令,可以查看前辈们使用的命令​。

​history

      从history命令历史中去揣摩命令,其实是一种乐趣,但是效率不高。再次分享一下前辈总结出来的命令清单

    ​    ​玩日志系统必备的17条操作命令|启动zookeeper kafka logstash storm 查看日志 我手上最珍贵的命令集

    ​测试结果压力一上去,w2wp.exe占用的cpu自己都能跑满100%,有没有filebeat,这cpu都能到100%,开着filebeat,filebeat能抢走40%到80%的cpu,随着日志压力大小会有所变化。

    ​2.4 完整的日志收集环境+模拟大量线上日志+Filebeat限制cpu

    ​当业务忙的时候,日志也多,这个时候,日志收集​就不要抢戏了。做为测试环境,配置​低,就限制1个cpu内核。开启压测后,显示Filebeat的cpu被限制在20%以下,不会出现Filebeat抢占过高的业务​资源。

​​小结:本次Filebeat占用CPU过高的case处理,最终的结论是max_procs: 1,为了这13个字符,设计了多种压测环境,回过头来想想,其实一开始就想限制cpu使用,万万没有想到是max_procs​。资源是有限的, cpu、内存,都是这样子​。嗯,cpu就这么控制一下​!​

作者:钟代麒

出处:http://www.jishudao.com/
版权归作者所有,转载请注明出处