索引超限问题 by Scott-YuYan · Pull Request #9 · hcsp/multithread-word-count

Scott-YuYan · 2019-09-14T06:19:29Z

有个索引超限问题不知道怎么解决

hcsp-bot · 2019-09-14T06:19:34Z

🎉 感谢提交Pull Request！请稍等片刻，我们已经将其提交到CI进行检查，一旦有结果会立即通知您！
不过，我们发现你在这个仓库中还打开了其他的Pull Request：

#8

我们不鼓励同时打开多个Pull Request，请集中精力于现在的这个Pull Request，谢谢！:pray:

hcsp-bot · 2019-09-14T06:20:04Z

你的提交 ed04930 ，似乎失败了：Your tests failed on CircleCI

😅 请不要气馁，仔细分析原因，再接再厉！

blindpirate · 2019-09-14T06:22:54Z

src/main/java/com/github/hcsp/multithread/WordCount.java

@@ -1,14 +1,77 @@
 package com.github.hcsp.multithread;

+import com.sun.deploy.security.SelectableSecurityManager;


这是什么鬼

blindpirate · 2019-09-14T06:23:33Z

src/main/java/com/github/hcsp/multithread/WordCount.java

+        List<Future<Map<String, Integer>>> list = new ArrayList<>();
+        Map<String, Integer> finalResult = new HashMap<>();
+        for (int i = 0; i < threadNum; ++i) {
+            while (files.get(0).exists()) {


这里如果files为空呢？

Suggested change

while (files.get(0).exists()) {

while (!files.isEmpty()) {

我这样建议只是指出你的问题，并不代表这样改了你的程序就能正常工作了。你最好避免多个子线程同时操作List，原因是：

你的List没有同步。

你怎么保证List是可以修改的，就调用它的remove方法？

很容易出线程安全问题。

blindpirate · 2019-09-14T06:27:57Z

src/main/java/com/github/hcsp/multithread/WordCount.java

+                        }
+                    }
+                    files.remove(0);
+                    threadPool.shutdown();


每个子线程执行完就自己把自己的线程池关掉？这个操作很骚啊。

虽然你从线程池中的线程里关闭肯定会遇到异常，但是我还是想问，线程1执行完了就把线程池整个关掉，那别的线程要是还在执行咋办？

blindpirate

最后的一个建议是，别写这么长的方法，因为很难理解，人类的思维可能跟不上。我的建议是，你声明一个这样的内部类：

static class Worker implements Callable<Map<String, Integer>> {
    List<File> 子线程分配到的一两个文件;
    @Override
    ...
}


// 在count方法中
// 先把文件列表按照线程数量切割成若干个子列表
threadPool.submit(new Worker(这个子线程分配到的文件子列表))

Scott-YuYan · 2019-09-14T06:31:26Z

好的，谢谢老师

…

------------------ 原始邮件 ------------------ 发件人: "Bo Zhang"<notifications@github.com>; 发送时间: 2019年9月14日(星期六) 下午2:30 收件人: "hcsp/multithread-word-count"<multithread-word-count@noreply.github.com>; 抄送: "MR.mister"<845058547@qq.com>; "Author"<author@noreply.github.com>; 主题: Re: [hcsp/multithread-word-count] 索引超限问题 (#9) @blindpirate commented on this pull request. 最后的一个建议是，别写这么长的方法，因为很难理解，人类的思维可能跟不上。我的建议是，你声明一个这样的内部类： static class Worker implements Callable<Map<String, Integer>> { List<File> 子线程分配到的一两个文件; } — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Scott-YuYan · 2019-09-14T06:38:49Z

应该放最外面去的

…

------------------ 原始邮件 ------------------ 发件人: "Bo Zhang"<notifications@github.com>; 发送时间: 2019年9月14日(星期六) 下午2:37 收件人: "hcsp/multithread-word-count"<multithread-word-count@noreply.github.com>; 抄送: "MR.mister"<845058547@qq.com>; "Author"<author@noreply.github.com>; 主题: Re: [hcsp/multithread-word-count] 索引超限问题 (#9) @blindpirate commented on this pull request. In src/main/java/com/github/hcsp/multithread/WordCount.java: + Map<String, Integer> finalResult = new HashMap<>(); + for (int i = 0; i < threadNum; ++i) { + while (files.get(0).exists()) { + File file = files.get(0); + list.add(threadPool.submit(() -> { + Map<String, Integer> result = new HashMap<>(); + List<String> oneFileToString = Files.readAllLines(file.toPath()); + for (String content : oneFileToString) { + String[] contentToArray = content.split(" "); + for (String element : contentToArray + ) { + result.put(element, result.getOrDefault(element, 0) + 1); + } + } + files.remove(0); + threadPool.shutdown(); 虽然你从线程池中的线程里关闭肯定会遇到异常，但是我还是想问，线程1执行完了就把线程池整个关掉，那别的线程要是还在执行咋办？ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

hcsp-bot · 2019-09-15T02:14:18Z

你的提交 ee933fd ，似乎失败了：Your tests failed on CircleCI

😅 请不要气馁，仔细分析原因，再接再厉！

Scott-YuYan · 2019-09-15T02:16:22Z

src/main/java/com/github/hcsp/multithread/WordCount.java

+
+            for (Future<Map<String, Integer>> future : list
+            ) {
+                Map<String, Integer> resultFromThread = future.get();


@blindpirate 老师，我调试到这里的时候，发现就只提示了一个 application is running了。想问下是为什么呢？

你难道没注意到你的电脑CPU负载很高么？所有的线程都在跑死循环啊。。。

Scott-YuYan · 2019-09-15T02:17:10Z

src/main/java/com/github/hcsp/multithread/WordCount.java

+        }
+    }
+
+    private synchronized static List<List<File>> averageList(List<File> files, Integer number) {


@blindpirate 这个文件拆分的方法还有些问题

我有没有跟你说过，不要自己造轮子？自己尝试搜索一下有没有类似方法？如果你需要的话，我可以允许你修改pom.xml.

Scott-YuYan · 2019-09-15T02:22:11Z

src/main/java/com/github/hcsp/multithread/WordCount.java

+        public Map<String, Integer> call() throws Exception {
+            Map<String, Integer> result = new ConcurrentHashMap<>();
+            while (!files.isEmpty()) {
+                for (File file : files) {


这个方法的时间复杂度是太高了，最开始的时候我是用files.remove(index)读取list中的文件的，会报出ConcurrentModificationException，尝试了加锁，已经copyOnWriteArrayList几种方法都没效。

这个时间复杂度有啥高的？

blindpirate · 2019-09-15T02:33:25Z

src/main/java/com/github/hcsp/multithread/WordCount.java

+        @Override
+        public Map<String, Integer> call() throws Exception {
+            Map<String, Integer> result = new ConcurrentHashMap<>();
+            while (!files.isEmpty()) {


把这个while去掉就好了。不明白为啥你要套两层循环：

while

for (File file : files)

因为files永远不为空，所以这个循环永远不会结束。

Scott-YuYan · 2019-09-15T02:42:37Z

gvava里面好像有一个，那我可以引用了

…

------------------ 原始邮件 ------------------ 发件人: "Bo Zhang"<notifications@github.com>; 发送时间: 2019年9月15日(星期天) 上午10:37 收件人: "hcsp/multithread-word-count"<multithread-word-count@noreply.github.com>; 抄送: "MR.mister"<845058547@qq.com>; "Author"<author@noreply.github.com>; 主题: Re: [hcsp/multithread-word-count] 索引超限问题 (#9) @blindpirate commented on this pull request. In src/main/java/com/github/hcsp/multithread/WordCount.java: + } + } + } + return result; + } + } + + private void mergeResultFromThread(Map<String, Integer> resultFromThread, + Map<String, Integer> finalResult) { + for (Map.Entry<String, Integer> entry : resultFromThread.entrySet()) { + int resultNumber = finalResult.getOrDefault(entry.getKey(), 0) + entry.getValue(); + finalResult.put(entry.getKey(), resultNumber); + } + } + + private synchronized static List<List<File>> averageList(List<File> files, Integer number) { 我有没有跟你说过，不要自己造轮子？自己尝试搜索一下有没有类似方法？如果你需要的话，我可以允许你修改pom.xml. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

hcsp-bot · 2019-09-15T03:08:36Z

你的提交 a219aab ，似乎失败了：Your tests failed on CircleCI

😅 请不要气馁，仔细分析原因，再接再厉！

hcsp-bot · 2019-09-15T03:12:30Z

你的提交 7dd9a6c ，似乎失败了：Your tests failed on CircleCI

😅 请不要气馁，仔细分析原因，再接再厉！

hcsp-bot · 2019-09-15T03:16:09Z

你的提交 8500bed ，似乎失败了：Your tests failed on CircleCI

😅 请不要气馁，仔细分析原因，再接再厉！

… into YY

hcsp-bot · 2019-09-15T03:34:18Z

恭喜你，你的提交 9ebc882 已经通过我们的CI检查：Your tests passed on CircleCI!

👍 它会被自动merge后revert。请不要骄傲，继续挑战！

This reverts commit 076dc0e.

blindpirate · 2019-09-15T07:34:09Z

src/main/java/com/github/hcsp/multithread/WordCount.java

 public class WordCount {
-    public WordCount(int threadNum) {}
+    private final int threadNum;
+    private ExecutorService threadPool;


所以这玩意为什么不是final的呢。。。

blindpirate · 2019-09-15T07:34:29Z

src/main/java/com/github/hcsp/multithread/WordCount.java

+
+        @Override
+        public Map<String, Integer> call() throws Exception {
+            Map<String, Integer> result = new ConcurrentHashMap<>();


其实这里就没必要用ConcurrentHashMap了，想一想为什么？

因为这是一个局部变量啊，没有任何并发写操作发生啊。

Scott-YuYan · 2019-09-15T07:36:36Z

因为已经加锁了吧

…

---原始邮件--- 发件人: "Bo Zhang"<notifications@github.com> 发送时间: 2019年9月15日(星期日) 下午3:34 收件人: "hcsp/multithread-word-count"<multithread-word-count@noreply.github.com>; 抄送: "JiangXi-YuYan"<845058547@qq.com>;"Author"<author@noreply.github.com>; 主题: Re: [hcsp/multithread-word-count] 索引超限问题 (#9) @blindpirate commented on this pull request. In src/main/java/com/github/hcsp/multithread/WordCount.java: > + System.out.println(finalResult); + threadPool.shutdown(); + return finalResult; + } + } + + static class Worker implements Callable<Map<String, Integer>> { + List<File> files; + + Worker(List<File> files) { + this.files = files; + } + + @OverRide + public Map<String, Integer> call() throws Exception { + Map<String, Integer> result = new ConcurrentHashMap<>(); 其实这里就没必要用ConcurrentHashMap了，想一想为什么？ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

索引超限问题

ed04930

blindpirate reviewed Sep 14, 2019

View reviewed changes

Scott-YuYan added 2 commits September 15, 2019 10:11

Merge remote-tracking branch 'origin/master' into YY

fa8722a

调了好久，还是不知道怎么解决当前的问题。

ee933fd

Scott-YuYan commented Sep 15, 2019

View reviewed changes

blindpirate reviewed Sep 15, 2019

View reviewed changes

实æ�正了bug

a219aab

去掉不该提交的提交

7dd9a6c

mvn compile是通过的

8500bed

Scott-YuYan added 2 commits September 15, 2019 11:25

使用guava的List.partition分割List

acf3563

Merge branch 'master' of https://github.com/hcsp/multithread-word-count…

9ebc882

… into YY

hcsp-bot merged commit 076dc0e into master Sep 15, 2019

hcsp-bot added a commit that referenced this pull request Sep 15, 2019

Revert "索引超限问题 (#9)"

290dd86

This reverts commit 076dc0e.

blindpirate reviewed Sep 15, 2019

View reviewed changes

		@@ -1,14 +1,77 @@
		package com.github.hcsp.multithread;

		import com.sun.deploy.security.SelectableSecurityManager;

Conversation

Scott-YuYan commented Sep 14, 2019

Uh oh!

hcsp-bot commented Sep 14, 2019

Uh oh!

hcsp-bot commented Sep 14, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blindpirate left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Scott-YuYan commented Sep 14, 2019 via email

Uh oh!

Scott-YuYan commented Sep 14, 2019 via email

Uh oh!

hcsp-bot commented Sep 15, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Scott-YuYan commented Sep 15, 2019 via email

Uh oh!

hcsp-bot commented Sep 15, 2019

Uh oh!

hcsp-bot commented Sep 15, 2019

Uh oh!

hcsp-bot commented Sep 15, 2019

Uh oh!

hcsp-bot commented Sep 15, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Scott-YuYan commented Sep 15, 2019 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

blindpirate left a comment •

edited

Loading