摘要: 原创出处 blog.csdn.net/qililong88/article/details/114320641 「qililong88」欢迎转载,保留摘要,谢谢!
有一个大List集合,遍历进行一些耗时操作,不能达到性能要求,查询日志,单个任务虽然有不少数据库和第三方API请求,比较耗时,但返回效率尚可,所以优先采用多线程方式进行处理并行请求数据库和第三方API,因为处理完还要对list所属的数据进行操作,所以,线程池多线程处理要等待全部处理完。
相关的代码如下:
@Test public void testTB () { List < String > list = new ArrayList < > (); for (int i = 0 ; i < 900 ; i++) { list.add("a" ); } ExecutorService touchWorker = Executors.newFixedThreadPool(4 , new ThreadFactoryBuilder().setNameFormat("touch-send-worker-%d" ).build()); int size = list.size(); if (size > 100 ) { int batch = size % 100 == 0 ? size / 100 : size / 100 + 1 ; for (int j = 0 ; j < batch; j++) { int end = (j + 1 ) * 100 ; if (end > size) { end = size; } List < String > subList = list.subList(j * 100 , end); touchWorker.execute(() - > sleepTest(subList)); } touchWorker.shutdown(); while (true ) { if (touchWorker.isTerminated()) { break ; } } } else { sleepTest(list); } } private void sleepTest (List < String > subList) { for (String i: subList) { try { System.out.println("######" + i + "######" + Thread.currentThread().getName()); } catch (Exception e) { e.printStackTrace(); } } }
void shutdown()
启动一次顺序关闭,执行以前提交的任务,但不接受新任务。若已经关闭,则调用没有其他作用。
抛出:SecurityException
- 如果安全管理器存在并且关闭,此 ExecutorService
可能操作某些不允许调用者修改的线程(因为它没有保持RuntimePermission("modifyThread")
),或者安全管理器的 checkAccess
方法拒绝访问。
boolean isTerminated()
若关闭后所有任务都已完成,则返回true。注意除非首先调用shutdown
或shutdownNow
,否则isTerminated
永不为true。返回:若关闭后所有任务都已完成,则返回true。
当然还有一种方法,是之前写的,方法比上边的臃肿了,不过会获取返回结果进行处理:逻辑是获取所有页面的List,多线程遍历List后,将所有页面的违规词查出发送邮件,代码:
@Test public void getViolationsLandpageByThreadPool () { try { landPageService.getViolationsLandpageByThreadPool("1年" ); } catch (Exception e) { e.printStackTrace(); } } long start = System.currentTimeMillis();List<LandPage> list = landPageDao.getPageIdAndPath(); SensitiveWordInit sensitiveWordInit = new SensitiveWordInit(); Map sensitiveWordMap = sensitiveWordInit.initKeyWord(vioKey); SensitivewordEngine.sensitiveWordMap = sensitiveWordMap; int threadSize = 11000 ;int dataSize = list.size();int threadNum = dataSize / threadSize + 1 ;boolean special = dataSize % threadSize == 0 ;ExecutorService exec = Executors.newFixedThreadPool(threadNum); List<Callable<List<LandPage>>> tasks = new ArrayList<Callable<List<LandPage>>>(); Callable<List<LandPage>> task = null ; List<LandPage> cutList = null ; for (int i = 0 ; i < threadNum; i++) { if (i == threadNum - 1 ) { if (special) { break ; } cutList = list.subList(threadSize * i, dataSize); } else { cutList = list.subList(threadSize * i, threadSize * (i + 1 )); } final List<LandPage> listStr = cutList; task = new Callable<List<LandPage>>() { @Override public List<LandPage> call () throws Exception { List<LandPage> result = new ArrayList<LandPage>(); for (LandPage landPage : listStr) { Long pageId = landPage.getPageId(); String path = landPage.getPath(); Integer version = landPage.getVersion(); String pageUrl = landPage.getPageUrl(); String actualUser = landPage.getActualUser(); Integer pageType = landPage.getPageType(); if (StringUtils.isNotBlank(path)) { String html = httpRequest(path); if (StringUtils.isNotBlank(html)){ html = html.replaceAll("<!--(.*?)-->" ,"" ); Set<String> bufferSet = SensitivewordEngine.getSensitiveWord(html, 1 ); String[] word = { "一年" ,"1年学完" ,"一年学完" ,"1年内学完" ,"一年内学完" }; for (int i=0 ;i<word.length;i++){ if (html.contains(word[i])){ bufferSet.add(word[i]); } } if (null !=bufferSet&&bufferSet.size()>0 ) { String sensitiveWord = bufferSet == null ? null : bufferSet.toString(); if ("[]" .equals(sensitiveWord)){ sensitiveWord = "" ; } LandPage page = new LandPage(); page.setPageId(pageId); page.setPath(path); page.setVersion(version); page.setDescription(sensitiveWord); page.setPageUrl(pageUrl); page.setActualUser(actualUser); page.setPageType(pageType); result.add(page); System.out.println(pageUrl); } } } } return (List<LandPage>) result; } }; tasks.add(task); } List<Future<List<LandPage>>> results = exec.invokeAll(tasks); List<LandPage> result = new ArrayList<LandPage>(); for (Future<List<LandPage>> future : results) { result.addAll(future.get()); } exec.shutdown(); System.out.println("线程任务执行结束" ); System.err.println("执行任务消耗了 :" + (System.currentTimeMillis() - start) + "毫秒" ); System.out.println("共有###########" +list.size() );
result就是需要发送邮件的相关结果了