Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
C
crawler_store
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
litianle
crawler_store
Commits
b9984a89
Commit
b9984a89
authored
Mar 20, 2022
by
litianle
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
添加大润发
parent
7e3262e5
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
69 additions
and
1 deletion
+69
-1
RTMartCrawler.java
...deeprec/crawler/modules/RTMart/crawler/RTMartCrawler.java
+68
-0
SevenElevenCrawler.java
...wler/modules/seven_Eleven/crawler/SevenElevenCrawler.java
+1
-1
No files found.
src/main/java/com/deeprec/crawler/modules/RTMart/crawler/RTMartCrawler.java
0 → 100644
View file @
b9984a89
package
com
.
deeprec
.
crawler
.
modules
.
RTMart
.
crawler
;
import
cn.hutool.http.HttpUtil
;
import
cn.hutool.json.JSONObject
;
import
com.deeprec.crawler.entry.CrawlerStore
;
import
com.deeprec.crawler.mapper.InsertDB
;
import
lombok.extern.slf4j.Slf4j
;
import
org.jsoup.Jsoup
;
import
org.jsoup.nodes.Document
;
import
org.jsoup.nodes.Element
;
import
org.jsoup.select.Elements
;
@Slf4j
public
class
RTMartCrawler
{
private
static
final
String
SK_NAME
=
"大润发"
;
private
static
final
String
URL
=
"https://www.rt-mart.com.cn/stores/store?size=1200"
;
public
static
void
main
(
String
[]
args
)
{
String
response
=
HttpUtil
.
createGet
(
"https://www.rt-mart.com.cn/stores/store"
)
.
form
(
"size"
,
1200
)
.
execute
().
body
();
Document
document
=
Jsoup
.
parse
(
response
);
Element
stores
=
document
.
getElementsByClass
(
"content-store__list"
).
get
(
0
);
Elements
storeList
=
stores
.
getElementsByClass
(
"store-right"
);
for
(
Element
store
:
storeList
)
{
Element
a
=
store
.
getElementsByTag
(
"a"
).
get
(
0
);
String
name
=
a
.
text
();
Element
storeInfo
=
store
.
getElementsByClass
(
"store-right__info"
).
get
(
0
);
Elements
ps
=
storeInfo
.
getElementsByTag
(
"p"
);
String
openingHours
=
""
,
address
=
""
,
serviceTel
=
""
,
groupTel
=
""
;
for
(
Element
p
:
ps
)
{
String
[]
split
=
p
.
text
().
split
(
":"
);
if
(
split
[
0
].
equals
(
"营业时间"
))
{
openingHours
=
split
[
1
];
}
if
(
split
[
0
].
equals
(
"门店地址"
))
{
address
=
split
[
1
];
}
if
(
split
[
0
].
equals
(
"服务电话"
))
{
serviceTel
=
split
[
1
];
}
if
(
split
[
0
].
equals
(
"团购电话"
))
{
groupTel
=
split
[
1
];
}
}
String
storeBus
=
""
;
if
(
storeInfo
.
getElementsByTag
(
"a"
).
size
()
!=
0
)
{
storeBus
=
URL
+
storeInfo
.
getElementsByTag
(
"a"
).
get
(
0
).
attr
(
"href"
);
}
CrawlerStore
crawlerStore
=
new
CrawlerStore
().
setCrawlerUrl
(
URL
)
.
setSkName
(
SK_NAME
)
.
setPhone
(
serviceTel
)
.
setName
(
name
)
.
setAddress
(
address
)
.
setExtInfo
(
new
JSONObject
()
.
set
(
"openingHours"
,
openingHours
)
.
set
(
"groupTel"
,
groupTel
)
.
set
(
"storeBus"
,
storeBus
));
InsertDB
.
getDB
().
insert
(
crawlerStore
);
log
.
info
(
"区域: {},{}"
,
crawlerStore
.
getAddress
(),
crawlerStore
);
}
log
.
info
(
"大润发全部插入完成"
);
}
}
src/main/java/com/deeprec/crawler/modules/seven_Eleven/crawler/SevenElevenCrawler.java
View file @
b9984a89
...
@@ -43,7 +43,7 @@ public class SevenElevenCrawler {
...
@@ -43,7 +43,7 @@ public class SevenElevenCrawler {
.
setExtInfo
(
new
JSONObject
()
.
setExtInfo
(
new
JSONObject
()
.
set
(
"openDate"
,
store
.
getOpenDate
())
.
set
(
"openDate"
,
store
.
getOpenDate
())
.
set
(
"cityFullName"
,
store
.
getCityFullName
()));
.
set
(
"cityFullName"
,
store
.
getCityFullName
()));
//
InsertDB.getDB().insert(crawlerStore);
InsertDB
.
getDB
().
insert
(
crawlerStore
);
log
.
info
(
"区域: {},{}"
,
crawlerStore
.
getAddress
(),
crawlerStore
);
log
.
info
(
"区域: {},{}"
,
crawlerStore
.
getAddress
(),
crawlerStore
);
}
}
}
}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment